Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  nodelist  faq  login

Someone is speaking well of you.


rocksolid / Tor / None

SubjectAuthor
* Very good research on detection of hidden servicesAnonymous
+- Re: Very good research on detection of hidden servicesRetro Guy
+- Re: Very good research on detection of hidden servicesAnonymous
+- Re: Very good research on detection of hidden servicesAnonymous
+- Re: Very good research on detection of hidden servicesAnonymous
+* Re: Very good research on detection of hidden servicesAnonymous
|`- Re: Very good research on detection of hidden servicesRetro Guy
+- Re: Very good research on detection of hidden servicesAnonymous
+- Re: Very good research on detection of hidden servicesAnonymous
+- Re: Very good research on detection of hidden servicesAnonymous
+- >Re: Very good research on detection of hidden servicesAnonymous
+* Final stats and address listAnonymous
|`- Re: Final stats and address listAnonymous
`- NoneAnonymous

1
Subject: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Mon, 29 Mar 2021 16:19 UTC
Path: i2pn2.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Very good research on detection of hidden services
Date: Mon, 29 Mar 2021 09:19:01 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.482.199atu@anon.com>
Content-Type: text/plain; charset=UTF-8
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="8865"; mail-complaints-to="usenet@novabbs.org"
View all headers
https://journals.riverpublishers.com/index.php/JCSANDM/article/view/1141/651

Also uploaded here:
-------------------------------------------------------------------------------------------------------------
Filenames: Detecting_and_tracking_onion_services_1141-Research_Article-4875-1-10-20200125.pdf
Download link:
http://ana5icnp3au4qy5i.onion/?1srAyikDSYNg1tgEt75t3pFKNp1
-------------------------------------------------------------------------------------------------------------

The study is recent, and it shows (among many interesting details) two things:
1) the actual number of tor servers actually offering some useful content is still relatively small. We are a small community, despite all the media buzz.
2) a vast majority of the top 10 requested services are really drug markets.

--
Posted on def2


Subject: Re: Very good research on detection of hidden services
From: Retro Guy
Newsgroups: rocksolid.shared.tor
Organization: Rocksolid Light
Date: Tue, 30 Mar 2021 05:42 UTC
References: 1
Path: i2pn2.org!.POSTED!not-for-mail
From: retro...@novabbs.com (Retro Guy)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Mon, 29 Mar 2021 22:42:06 -0700
Organization: Rocksolid Light
Message-ID: <20210329224206.82305712ca158b01b3dd92ab@novabbs.com>
References: <to.482.199atu@anon.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Info: i2pn2.org; posting-account="retrobbs1";
logging-data="28779"; mail-complaints-to="usenet@i2pn2.org"
X-Newsreader: Sylpheed 3.5.0beta1 (GTK+ 2.24.25; x86_64-pc-linux-gnu)
View all headers
On Mon, 29 Mar 2021 09:19:01 -0700
Anonymous <poster@anon.com> wrote:

https://journals.riverpublishers.com/index.php/JCSANDM/article/view/1141/651

Also uploaded here:
-------------------------------------------------------------------------------------------------------------
Filenames: Detecting_and_tracking_onion_services_1141-Research_Article-4875-1-10-20200125.pdf
Download link:
http://ana5icnp3au4qy5i.onion/?1srAyikDSYNg1tgEt75t3pFKNp1

Similar to how cornering a service in Freenet is done also:
" We  place  a
set  of  20  volunteer  servers  on  the  Tor  network  in  order  to  collect  .onion
addresses."



Subject: Re: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Sat, 3 Apr 2021 13:30 UTC
References: 1
Attachments: onion_spider_0.1.zip (application/zip)
Path: i2pn2.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Sat, 03 Apr 2021 06:30:14 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.484.4eihky@anon.com>
References: <to.482.199atu@anon.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=fea5e0bf9aaaced81d01bdd26b5421ecef56073a
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="26626"; mail-complaints-to="usenet@novabbs.org"
View all headers
I got curious now, so I coded a little spider script to discover onion sites. Underneath you can find the code, I also attach an archive in case it gets messed up. Current features of the spider:
-collects v2/v3 addresses of hidden services that host webservers
-collects all links found on any visited site
-single-threaded (because I am not after speed, and I don't want to ddos the tor network or any site)
-gives status update after each run (statistics)
-uses lynx for retrieval  of the sites and all the messy html parsing and extraction of the links (-dump -listonly)
-one run = one link only --> process can be interrupted at any time without data loss
-visits all valid http/https-links on onion domains, while trying to avoid any file download (images, pdfs, ...)
-uses a 2 step random choice algorithm to select the next link to visit, as a result pages with a lot of links have the same chance to get chosen as pages with one link only.

I might add other features later, like link chains (this would allow to make a mapping of the existing services and how they link to each other).
Of course, such programs already exist. and more sophisticated, faster, etc. This is more about learning, and about reaching the target with minimal resources. So, here it comes:

#!/bin/bash
###############################################################
################## v2/v3 scanner ##############################
# a little bash script to scan the darkweb for tor addresses
# adjust the proxy variables in /etc/lynx/lynx.cfg if needed
# than just call the script from a subdir, and let run
# you will need a file called "links.txt" containing onion
# links to start the script of (there are some popular
# addresses preloaded).
# the script runs in an endless loop until there are no more
# links to be followed. you can stop anytime with CTRL+C
# without loosing anything but the currently
# running request. Another start of the script, and it will
# pickup where it was interrupted. all of this is
# EXPERIMENTAL CODE, USE AT OWN RISK
###############################################################
counter=0
touch ./visited_links.txt
while true
do
counter=$(("$counter"+1))
sort -u links.txt | grep -E 'http:\/\/[[:alnum:]]{16,56}\.onion.*' | sort > links_new.txt && mv links_new.txt links.txt
visited_link_count=$(wc -l visited_links.txt | cut -f1 -d ' ')
link_count=$(wc -l links.txt | cut -f1 -d ' ')
service_count=$(grep -E -o 'http:\/\/[[:alnum:]]{16,56}\.onion' links.txt | sort -u | wc -l)
visited_service_count=$(grep -E -o 'http:\/\/[[:alnum:]]{16,56}\.onion' visited_links.txt | sort -u | wc -l)
echo -e " we have visited $visited_link_count links and discovered $visited_service_count onion addresses, we still have $link_count unique links on $service_count hosts\n\r"
service=$(grep -E -o 'http:\/\/[[:alnum:]]{16,56}\.onion.*' links.txt | sort -u | sort -R | head -n 1)
link=$(grep "$service" links.txt | sort -R | head -n 1)
if [ -z $link ]; then
echo "no more links, we are done"
exit 0
fi
echo -e "trying to get $link in run number $counter,\n\r"
echo "$link" >> visited_links.txt
sort -u visited_links.txt | sort > visited_links_new.txt && mv visited_links_new.txt visited_links.txt
comm -23 links.txt visited_links.txt > links_new.txt && mv links_new.txt links.txt
lynx -dump -listonly $link | grep -E 'http:\/\/[[:alnum:]]{16,56}\.onion.*' | grep -E -i -v '.*\.jpg$|.*\.gif$|.*\.png$|.*\.pdf$|.*\.mp3$|.*\.m3u$|.*\.avi$|.*\.jpeg$|.*\.bmp$|.*\.mkv$' | cut -f2- -d '.' | cut -c2- >> links.txt
done

Content of the file links.txt (seed file):

http://tor66sewebgixwhcqfnp5inzp5x5uohhdy3kvtnyfxc2e5mxiuh34iid.onion/
http://suprbayoubiexnmp.onion/
http://3bbad7fauom4d6sgppalyqddsqbf5u5p56b5k5uk2zxsy3d6ey2jobad.onion/
http://tordexu73joywapk2txdr54jed4imqledpcvcuf75qsas2gwdgksvnyd.onion/
http://torchdeedp3i2jigzjdmfpn5ttjhthh5wbmda2rr3jvqjg5p77c54dqd.onion/
http://zqktlwi4fecvo6ri.onion/wiki/index.php/Main_Page
http://phobosxilamwcg75xt22id7aywkzol6q6rfl2flipcqoc4e4ahima5id.onion/search?query=linklist

I might publish the addresses I found at one point.


Attachments: onion_spider_0.1.zip (application/zip)
Subject: Re: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Sun, 4 Apr 2021 15:57 UTC
References: 1
Attachments: onion_spider_0.11.zip (application/zip)
Path: i2pn2.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Sun, 04 Apr 2021 08:57:59 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.488.3v33te@anon.com>
References: <to.482.199atu@anon.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=72dcfc5232a13271cebdc48cdde42f944900ce98
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="21678"; mail-complaints-to="usenet@novabbs.org"
View all headers
fea5e0bf9aaaced81d
improved version 0.11 attached. new features:

-checks if host is up before attempting download, and logs hosts that are not reachable
-blocklist to prevent scanning specific addresses
-creates list with reachable hosts (addresses)
-adjustable limit for following links per host (default=5) for less stress on servers and faster progress

With the old version, I could discover ca. 1300 v2 and 1800 v3 addresses in a short time, only following the seeds published.


Attachments: onion_spider_0.11.zip (application/zip)
Subject: Re: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Sun, 4 Apr 2021 16:01 UTC
References: 1
Path: i2pn2.org!i2pn.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Sun, 04 Apr 2021 09:01:06 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.489.12leg@anon.com>
References: <to.482.199atu@anon.com>
Content-Type: text/plain; charset=UTF-8
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="29049"; mail-complaints-to="usenet@novabbs.org"
View all headers
72dcfc5232a13271ce
oh I forgot:

the links found on each host are now collected, so that they can be used later to create a linkmap of onion sites.

--
Posted on def2


Subject: Re: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Sun, 4 Apr 2021 20:44 UTC
References: 1
Attachments: onion_spider_0.12.zip (application/zip)
Path: i2pn2.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Sun, 04 Apr 2021 13:44:26 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.490.3zm5cr@anon.com>
References: <to.482.199atu@anon.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=0d18e40d7e341c430251bb3179b08bb338611dbe
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="7271"; mail-complaints-to="usenet@novabbs.org"
View all headers
72dcfc5232a13271ce
The amount of hidden services that I could identify so far is pretty close to the link lists of the excellent service
http://links.communzyxz3qfpum5tnvrfvvrr4jlosbq4mzeskigoionqqdylmlhmid.onion/
which lists some 3300 addresses. So I guess this is really the ballpark we are playing in here. onion-net is small indeed.
Of course there might be a lot more hidden services which are just known by some few, and that are not listed anywhere.
But for the public sites, I expected a bit more. Counting out all the scams, I guess i2p is mot much smaller today (I mean in terms of public sites).
Attached the latest version with a couple of bug fixes and a small new feature (seed addresses without download limit).
And btw, the tool should also work for i2p with some tweaks in the regexes for the links.


Attachments: onion_spider_0.12.zip (application/zip)
Subject: Re: Very good research on detection of hidden services
From: Retro Guy
Newsgroups: rocksolid.shared.tor
Organization: Rocksolid Light
Date: Mon, 5 Apr 2021 06:54 UTC
References: 1 2
Path: i2pn2.org!.POSTED!not-for-mail
From: retro...@novabbs.com (Retro Guy)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Sun, 4 Apr 2021 23:54:28 -0700
Organization: Rocksolid Light
Message-ID: <20210404235428.27b2699d54070c1e78d50897@novabbs.com>
References: <to.482.199atu@anon.com>
<to.490.3zm5cr@anon.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Info: i2pn2.org; posting-account="retrobbs1";
logging-data="14297"; mail-complaints-to="usenet@i2pn2.org"
X-Newsreader: Sylpheed 3.5.0beta1 (GTK+ 2.24.25; x86_64-pc-linux-gnu)
View all headers
On Sun, 04 Apr 2021 13:44:26 -0700
Anonymous <poster@anon.com> wrote:

72dcfc5232a13271ce
The amount of hidden services that I could identify so far is pretty close to the link lists of the excellent service
http://links.communzyxz3qfpum5tnvrfvvrr4jlosbq4mzeskigoionqqdylmlhmid.onion/
which lists some 3300 addresses. So I guess this is really the ballpark we are playing in here. onion-net is small indeed.
Of course there might be a lot more hidden services which are just known by some few, and that are not listed anywhere.
But for the public sites, I expected a bit more. Counting out all the scams, I guess i2p is mot much smaller today (I mean in terms of public sites).
Attached the latest version with a couple of bug fixes and a small new feature (seed addresses without download limit).
And btw, the tool should also work for i2p with some tweaks in the regexes for the links.

Very nice. Reading your bash scripts makes my brain hurt as I'm not much of a bash programmer lol. I didn't even realize you could build a website with bash until you did so with bbs.

These would be nice lists to see. Of course we don't know what the sites are but lists are always good.

Retro Guy


Subject: Re: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Mon, 5 Apr 2021 10:52 UTC
References: 1
Path: i2pn2.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Mon, 05 Apr 2021 03:52:16 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.494.48eg6t@anon.com>
References: <to.482.199atu@anon.com>
Content-Type: text/plain; charset=UTF-8
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="24287"; mail-complaints-to="usenet@novabbs.org"
View all headers
f8835fd8987e6f2469
Reading your bash scripts makes my brain hurt
There is some php code for this, I guess it is pretty good (judging by who wrote it):
https://github.com/DanWin/onion-link-list
The reason I use bash for this is just that it is really really simple and fast (for me), plus I like the minimalistic approach of shell scripts.

--
Posted on def2


Subject: Re: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Mon, 5 Apr 2021 15:12 UTC
References: 1
Attachments: onion_spider_0.16.zip (application/zip)
Path: i2pn2.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Mon, 05 Apr 2021 08:12:44 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.495.w0ctm@anon.com>
References: <to.482.199atu@anon.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=99223959cca9d2760c969f3af49070d60e449752
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="29220"; mail-complaints-to="usenet@novabbs.org"
View all headers
new version 0.16 attached. changes:
-some bug were fixed (yes, even in these few lines, if you can believe it)
-speed improvement by adding seeds
-speed improvement by biasing the random choice of the address to retrieve towards hosts that have been found up previously
-gives out agglomerated results in one file for later processing
-gives out status reports to log file
-analysis of links found per host:
1) all links that point to the same host (internal links)
2) all links that point to other onion addresses
3) links from other pages that point to the host

Unfortunately, these numbers are not very accurate fo rnow due to several factors, and are vastly different in particular cases (mostly false negative, so links are not found).
However, high numbers found help to identify:
-sites with a lot of content
-sites with a lot of links
-sites that are known by many

In later versions I will probably use this to prioritize the crawling even more.

To analyze the tor network efficiently the well known sites would also make good starting points.

Stats from a current run:
onion addresses found: 8590
hosts confirmed up: 1687
hosts appearing down: 1894
to check: 5009


Attachments: onion_spider_0.16.zip (application/zip)
Subject: Re: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Tue, 6 Apr 2021 20:28 UTC
References: 1
Attachments: onion_spider_0.18.zip (application/zip)
Path: i2pn2.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Re: Very good research on detection of hidden services
Date: Tue, 06 Apr 2021 13:28:11 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.496.1geo7l@anon.com>
References: <to.482.199atu@anon.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=510b77c340944597198b8c99198ee08af67d0018
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="15101"; mail-complaints-to="usenet@novabbs.org"
View all headers
99223959cca9d2760c
version 0.18 attached.
-bugfixes, including one bug that would let the script run forever
-proper directory structure
-updated blocklist
-collected links are scanned for keywords of unwanted categories, ranking is displayed with the results
-speed gain by working on all v3 addresses first (quicker confirmation of service, plus v3 addresses tend be up more than v2 addresses)

Current stats:
discovered addresses: 11391
confirmed up: 4885
appearing down: 3098


Attachments: onion_spider_0.18.zip (application/zip)
Subject: >Re: Very good research on detection of hidden services
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Thu, 8 Apr 2021 18:38 UTC
References: 1
Attachments: up_cleaned.txt (text/plain)
Path: i2pn2.org!rocksolid2!.POSTED.127.213.60.80!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: >Re: Very good research on detection of hidden services
Date: Thu, 08 Apr 2021 11:38:01 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.500.1re2is@anon.com>
References: <to.482.199atu@anon.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=4d798b339a846a9f0bbf4d77ade54bb46e2a919d
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.213.60.80";
logging-data="12333"; mail-complaints-to="usenet@novabbs.org"
View all headers
510b77c34094459719
Updated stats:
discovered addresses: 13173
confirmed up: 6506
appearing down: 5328

Some 500 addresses are blocked by now, mostly for suspected or confirmed hosting of cp.
They are not part of the stats or the list attached.


Attachments: up_cleaned.txt (text/plain)
Subject: Final stats and address list
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Sat, 17 Apr 2021 19:22 UTC
References: 1
Path: i2pn2.org!i2pn.org!rocksolid2!.POSTED.127.166.132.178!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Final stats and address list
Date: Sat, 17 Apr 2021 12:22:22 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.517.1jncx5@anon.com>
References: <to.482.199atu@anon.com>
Content-Type: text/plain; charset=UTF-8
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.166.132.178";
logging-data="20776"; mail-complaints-to="usenet@novabbs.org"
View all headers
So, these are the final figures:

addresses found total: 10242
addresses found to be up: 8512
addresses up w/o cp: 7309
out of which v2: 3555
out of which v3: 3754

The link to the list with all addresses found to be up can be found here:
http://pastebinkaav4f5x.onion/a62300f111fbb7e79f03a3d124b66631337e5186

Use at own risk, I tried to filter out the cp, but might have failed in some cases.

Quite a contrast to the research that triggered it. Easy to find addresses, and most of them appear to be up, too. Guess spidering is just superior to the method that was used.
There should be lists longer than that, but actually I have not found them yet.
I thought for a while to make this into a permanent service (onion site), but there should be enough link lists for onion sites already.

--
Posted on def2


Subject: Re: Final stats and address list
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: Rocksolid Light
Date: Sun, 18 Apr 2021 00:52 UTC
References: 1 2
Path: i2pn2.org!.POSTED.10.136.143.187!not-for-mail
From: Anonym...@news.novabbs.org (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: Re: Final stats and address list
Date: Sun, 18 Apr 2021 00:52:35 +0000
Organization: Rocksolid Light
Message-ID: <d579e04c8ab39c43b204c9b5e02efd64@news.novabbs.org>
References: <to.482.199atu@anon.com> <to.517.1jncx5@anon.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org; posting-account="retrobbs1"; posting-host="10.136.143.187";
logging-data="26861"; mail-complaints-to="usenet@i2pn2.org"
User-Agent: Rocksolid Light (news.novabbs.com/getrslight)
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on novabbs.org
X-Rslight-Site: $2y$10$KWWtTWeBdb3kRzqzeUe9ROfE9HzpIPRF.DnLvJfLonJ2ImLwq.ImW
View all headers
Anonymous wrote:

So, these are the final figures:

addresses found total: 10242
addresses found to be up: 8512
addresses up w/o cp: 7309
out of which v2: 3555
out of which v3: 3754

Just downloaded the list. I'm considering a script to pull the title from each site and add it to the file. Is this something easy to do when making the list in the first place, or is doing it after the first run better?
--
Posted on Rocksolid Light


Subject: None
From: Anonymous
Newsgroups: rocksolid.shared.tor
Organization: rocksolid2 (novabbs.org)
Date: Sun, 18 Apr 2021 11:51 UTC
References: 1
Path: i2pn2.org!rocksolid2!.POSTED.127.166.132.178!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.shared.tor
Subject: None
Date: Sun, 18 Apr 2021 04:51:21 -0700
Organization: rocksolid2 (novabbs.org)
Message-ID: <to.522.1j2olw@anon.com>
References: <to.482.199atu@anon.com>
Content-Type: text/plain; charset=UTF-8
Injection-Info: novabbs.org; posting-account="def2"; posting-host="127.166.132.178";
logging-data="21434"; mail-complaints-to="usenet@novabbs.org"
View all headers
23e71008f4848d7f0c
pull the title from each site and add it to the file.
yes, i had something similar in place (link analysis)-

Is this something easy to do when making the list in the first place, or is doing it after the first run better?

i would say it is slightly faster to do it right away, but it is easy enough to do after as well.

The easiest is probably to get the page source and then parse it with regular expressions. Like this:

lynx -source http://kyue3hfjhqxj3df4hox46itdyz7euah457ldtg4d23evfggumxw4rkid.onion/u/ | gawk -v IGNORECASE=1 -v RS='</title' 'RT{gsub(/.*<title[^>]*>/,"");print;exit}'

--
Posted on def2


1
rocksolid light 0.7.2
clearneti2ptor