Rocksolid Light

Welcome to Rocksolid Light

register   nodelist   faq  


rocksolid / rocksolid.nodes / Re: basics are working

SubjectAuthor
* hey, what do know ? vichan supports nntpanon
+* Re: hey, what do know ? vichan supports nntpanon
|`* Re: hey, what do know ? vichan supports nntpRetro Guy
| `* Re: hey, what do know ? vichan supports nntpanon
|  `* Re: hey, what do know ? vichan supports nntpRetro Guy
|   `* Re: hey, what do know ? vichan supports nntpanon
|    `* Re: hey, what do know ? vichan supports nntpanon
|     `* Re: hey, what do know ? vichan supports nntpRetro Guy
|      `* Re: hey, what do know ? vichan supports nntpanon
|       `* Re: hey, what do know ? vichan supports nntpanon
|        +* Re: hey, what do know ? vichan supports nntpanon
|        |`* Re: hey, what do know ? vichan supports nntpanonymous
|        | +* Re: hey, what do know ? vichan supports nntpRetro Guy
|        | |`* Re: hey, what do know ? vichan supports nntpanon
|        | | `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        | |  `* Re: hey, what do know ? vichan supports nntpanonymous
|        | |   `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        | |    `* Re: hey, what do know ? vichan supports nntpanonymous
|        | |     `* Re: hey, what do know ? vichan supports nntpanon
|        | |      +* Re: hey, what do you know: vichan supports nntpanon
|        | |      |+- Re: hey, what do you know: vichan supports nntpRetro Guy
|        | |      |`* Re: hey, what do you know: vichan supports nntpanon
|        | |      | `- Re: hey, what do you know: vichan supports nntpRetro Guy
|        | |      `- Re: hey, what do know ? vichan supports nntpRetro Guy
|        | `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |  `* Re: hey, what do know ? vichan supports nntpanonymous
|        |   `* Re: hey, what do know ? vichan supports nntpanonymous
|        |    `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     +* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |`* Re: hey, what do know ? vichan supports nntpanonymous
|        |     | `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |  `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |   `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |    `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |     `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |      `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |       `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |        `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |         `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |          `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |           `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |            `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |             `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |              `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |               `* Re: hey, what do know ? vichan supports nntpanon
|        |     |                `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |                 `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                  `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                   `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                    `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                     `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |                      `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                       +* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                       |+- Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                       |`- Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |                       `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |                        `- Re: hey, what do know ? vichan supports nntpanonymous
|        |     `- Re: hey, what do know ? vichan supports nntpRetro Guy
|        `- Re: hey, what do know ? vichan supports nntpRetro Guy
+- Re: hey, what do know ? vichan supports nntpanon
+* Re: hey, what do know ? vichan supports nntpanon
|`* Re: hey, what do know ? vichan supports nntpanonymous
| +- Re: hey, what do you know....anon
| `* Re: hey, what do know ? vichan supports nntpanonymous
|  `* Re: hey, what do know ? vichan supports nntpRetro Guy
|   `- Re: hey, what do know ? vichan supports nntpanon
+* basics are workingAnonymous
|`- Re: basics are workingRetro Guy
+* guess i will open for business soonAnonymous
|+* Re: guess i will open for business soonMei
||`- Re: guess i will open for business soonRetro Guy
|`- Re: guess i will open for business soonRetro Guy
+- opsec check is goodAnonymous
+- > goo to go.Anonymous
+* yes that should workAnonymous
|`- Re: yes that should workRetro Guy
+* cron or not ?Anonymous
|`- Re: cron or not ?Retro Guy
+- themes on def2.i2pAnonymous
`* ok, ok, i switched the default themeAnonymous
 `* Re: ok, ok, i switched the default themeMei
  `- Re: ok, ok, i switched the default themeanon

Subject: Re: hey, what do know ? vichan supports nntp
From: anon@anon.com (anon)
Newsgroups: rocksolid.nodes
Organization: def5
Date: Sun, 24 May 2020 16:22 UTC

to dig out the thread starting message, this code should do it, right ? (from article-flat.php):

if($message) {
  // load thread-data and get IDs of the actual subthread
  $thread=thread_load($group);
  $subthread=thread_getsubthreadids($message->header->id,$thread);
  if($thread_articles == false) {
    sort($subthread);
  }

trw

Posted on def4


Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Sun, 24 May 2020 21:23 UTC
I did my own code to do the threading like vichan expects it, wasn't so hard after all (one recursive function, basically).
I just redo the message import, still get mistakes, but a lot less. seems more and more usable. If only I knew what vichan is expecting exactly as a date (this is the last (obvious) thing to solve).

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion


Subject: Re: hey, what do you know....
From: anon@anon.com (anon)
Newsgroups: rocksolid.nodes
Organization: def5
Date: Sun, 24 May 2020 23:38 UTC

If only I knew what vichan is expecting exactly as a date (this is the last (obvious) thing to solve).

Turns out that it expected what I had been trying at the very start: a unix timestamp. ...it just doesn't respect it !
So for some reason the date of the import is always displayed as the messages time, no matter what the original time is... which is no problem if updates happen in timely manner, but for rebuilding an existing message base it is not so good (at least one context of the messages is gone).
guess the image boards have always been fixated on short term, low latency interaction...

Posted on def4


Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Mon, 25 May 2020 21:49 UTC
had to rewrite the re-indexing code due to some bugs, this is the data exchange format I use now:

---snip

 "<04a2efaf39f90b7fd445b3d34b04e849$1@retrobbs.rocksolidbbs.com>": {
        "parents": [              "<5fa5f7ef9a71f09f4a486c3d579083bf@def4.com>",
            "<fb56c3093f6c0f918ca58cd34ad42065@def4.com>",               "<prlg96$jgf$1@def3.retrobbs.com>"
        ],
        "op": false,
        "ref": "<prlg96$jgf$1@def3.retrobbs.com>",
        "ref_org": "<5fa5f7ef9a71f09f4a486c3d579083bf@def4.com>",
        "solved": true
    },
    "<d11adec78757e4dad3e4b7d2f46880f5$1@dkzerogt6z6ybhcj.onion>": {
        "parents": [],
        "op": true      },
    "<b5d27591156eeb51329e572177153951$1@www.novabbs.com>": {
        "parents": [],
        "op": true
    },
    "<7c83a8ef135c1022d441adde9029441b$1@www.novabbs.com>": {
        "parents": [
            "<b5d27591156eeb51329e572177153951$1@www.novabbs.com>"
        ],
        "op": false,
        "ref": "<b5d27591156eeb51329e572177153951$1@www.novabbs.com>",
        "ref_org": "<b5d27591156eeb51329e572177153951$1@www.novabbs.com>",
        "solved": true
    },

---snap

with this implemented properly, most of the threads should work on vichan as well. than i just have to reindex all messages.
but first the rslight upgrade.

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion


Subject: Re: hey, what do know ? vichan supports nntp
From: Retro Guy@rslight.i2p (Retro Guy)
Newsgroups: rocksolid.nodes
Organization: Rocksolid Light
Date: Tue, 26 May 2020 09:07 UTC
anonymous wrote:

had to rewrite the re-indexing code due to some bugs, this is the data exchange format I use now:

---snip

with this implemented properly, most of the threads should work on vichan as well. than i just have to reindex all messages.
but first the rslight upgrade.

This is awesome! You're really making some great mods to vichan!

Once you have some code you want applied to spoolnews let me know. I'd like to either get most of it into a function (or two), or a script itself (vichan.php?)

I'll just put a toggle in config like I do with synchronet ($synchronet=true/false).

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com



Subject: Re: hey, what do know ? vichan supports nntp
From: anon@anon.com (anon)
Newsgroups: rocksolid.nodes
Organization: def5
Date: Tue, 26 May 2020 10:50 UTC

This is awesome! You're really making some great mods to vichan!

Thanks for the kind words, it is really fun to work on that.

Just in case some vichan or nntpchan dev should ever read this thread: I'm still standing on the proverbial "shoulders of giants". Anyway, thanks for this nice piece of software.

cheers

trw

Posted on def4


Subject: basics are working
From: poster@anon.com (Anonymous)
Newsgroups: rocksolid.nodes
Organization: i2pn2 (i2pn.org)
Date: Tue, 26 May 2020 15:34 UTC
Attachments: unnamed (image/png)
with some 11 exceptions my routine could glue all the threads together again.
this is still far from being elegant, mind you....

this code handles all the exporting from rslight to vichan.
-it checks the articles in the spool,
-builds an index file and a watermark file (if necessary),
-then resolves all the broken threads (fixed 10 iterations, have to make that dynamic later),
-then exports everything to vichan.

on the vichan side there is a modified post.php to get the messages.

so far the three parts (rslight, vichan and the script) are fully decoupled and individually called from cron.

in the long run code could be merged of course.

import script:

<?php

include "config.inc.php";
include "$file_newsportal";
include 'message.inc.php';

$groups = array(
'rocksolid.nodes',
'rocksolid.nodes.help',
'rocksolid.nodes.announce',
'rocksolid.shared.encryption',
'rocksolid.shared.entertainment',
'rocksolid.shared.freenet',
'rocksolid.shared.general',
'rocksolid.shared.hacking',
'rocksolid.shared.helpdesk',
'rocksolid.shared.i2p',
'rocksolid.shared.linux',
'rocksolid.shared.news',
'rocksolid.shared.offtopic',
'rocksolid.programming',
'rocksolid.shared.rocksolid',
'rocksolid.shared.security',
'rocksolid.social',
'rocksolid.shared.test',
'rocksolid.shared.tor',
);

if(!file_exists('xt.json')){
echo("no index found, building it \n\r");
$threads = build_threads($groups);
} else {
$threads = json_decode(file_get_contents('xt.json'), true);
$watermarks = json_decode(file_get_contents('xw.json'), true);
$threads = update_threads($groups, $threads, $watermarks);
}
$thread_keys = array_keys($threads);
for ($i = 0; $i < 10; $i++){
$threads = stage($thread_keys, $threads);
}

$jsonFile = fopen('xt.json', 'w');
fwrite($jsonFile, json_encode($threads, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonFile);

update_sync($threads);

exit;


function build_threads($groups){
$threads = array();
$watermarks = array();
foreach($groups as $group){
$watermarks[$group] = array();
echo("building index for group " . $group . "\n\r");
for ($i = 0; $i < 1000; $i++){
$message = message_read($i, 0, $group);
if($message){
echo("processing message # " . $i . " from group " . $group . "\r");
$rawData = json_decode(json_encode((array) $message, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES), true);
$ID = $rawData['header']['id'];
$Ref = $rawData['header']['references']['0'];
$firstRef = explode("<", $Ref);
$Ref = ("<" . $firstRef[1]);
$threads[$ID] = array();
$threads[$ID]['parents'] = array();
$threads[$ID]['synced'] = false;
$threads[$ID]['number'] = $i;
$threads[$ID]['group'] = $group;
$threads[$ID]['message'] = array();
$threads[$ID]['message'] = $rawData;
if($Ref == "<"){
$threads[$ID]['op'] = true;
$threads[$ID]['solved'] = true;
} else {
$threads[$ID]['op'] = false;
$threads[$ID]['ref'] = $Ref;
$threads[$ID]['ref_org'] = $rawData['header']['references']['0'];
$threads[$ID]['solved'] = false;
}
array_push($watermarks[$group],$i);
}
}
}
$jsonFile = fopen('xt.json', 'w');
fwrite($jsonFile, json_encode($threads, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonFile);
$jsonWatermarks = fopen('xw.json', 'w');
fwrite($jsonWatermarks, json_encode((array) $watermarks, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonWatermarks);
return $threads;
}
function update_threads($groups, $threads, $watermarks){
$update = false;
foreach($groups as $group){
echo("updating index for group " . $group . "\n\r");
for ($i = 0; $i < 1000; $i++){
if(!in_array($i,$watermarks[$group])){
$message = message_read($i, 0, $group);
if($message){
echo("processing message # " . $i . " from group " . $group . "\r");
$update = true;
$rawData = json_decode(json_encode((array) $message, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES), true);
$ID = $rawData['header']['id'];
$Ref = $rawData['header']['references']['0'];
$firstRef = explode("<", $Ref);
$Ref = ("<" . $firstRef[1]);
$threads[$ID] = array();
$threads[$ID]['parents'] = array();
$threads[$ID]['synced'] = false;
$threads[$ID]['number'] = $i;
$threads[$ID]['group'] = $group;
$threads[$ID]['message'] = array();
$threads[$ID]['message'] = $rawData;
if($Ref == "<"){
$threads[$ID]['op'] = true;
$threads[$ID]['solved'] = true;
} else {
$threads[$ID]['op'] = false;
$threads[$ID]['ref'] = $Ref;
$threads[$ID]['ref_org'] = $rawData['header']['references']['0'];
$threads[$ID]['solved'] = false;
}
array_push($watermarks[$group],$i);
}
}
}
}
if($update){
echo("saving updated threading\n\r");
$jsonFile = fopen('xt.json', 'w');
fwrite($jsonFile, json_encode($threads, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonFile);
echo("saving updated watermarks\n\r");
$jsonWatermarks = fopen('xw.json', 'w');
fwrite($jsonWatermarks, json_encode((array) $watermarks, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonWatermarks);
} else {
echo("no new files found\n\r");
}
return $threads;
}

function stage($thread_keys, $threads){
foreach($thread_keys as $thread_key){
if( ($threads[$thread_key]['solved'] !== true) && (threads[$thread_key]['op'] !== true) ){
echo("checking message" . $thread_key . "\r");
if ($threads[$thread_key]['parents']){
$to_check = end($threads[$thread_key]['parents']);
if(!array_key_exists($to_check, $threads)){
$threads[$thread_key]['op'] = true;
$threads[$thread_key]['ref'] = array();
$threads[$thread_key]['missing_ref'] = $to_check;
$threads[$thread_key]['solved'] = true;
} else {
if($threads[$to_check]['op'] == true){
$threads[$thread_key]['ref'] = $to_check;
$threads[$thread_key]['solved'] = true;
} else {
array_push($threads[$thread_key]['parents'], $threads[$to_check]['ref']);
array_filter($threads[$thread_key]['parents'], static function($var){return $var !== null;} );
}
}
} elseif ($threads[$thread_key]['ref']){
array_push($threads[$thread_key]['parents'], $threads[$thread_key]['ref']);
}
}
}
return $threads;
}

function update_sync($threads){
$update = false;
$thread_keys = array_keys($threads);
foreach($thread_keys as $thread_key){
if( ($threads[$thread_key]['synced'] == false) && ($threads[$thread_key]['solved'] == true) ){
echo("processing message # " . $threads[$thread_key]['number'] . " from group " . $threads[$thread_key]['group'] . "\n\r");
$rawData = $threads[$thread_key]['message'];
if($threads[$thread_key]['ref']){
$rawData['header']['references']['0'] = $threads[$thread_key]['ref'];
} else {
unset($rawData['header']['references']);
}
$paddedI = str_pad($threads[$thread_key]['number'], 5, '0', STR_PAD_LEFT);
$jsonFile = fopen('/var/www/html/vi/vichan-master/debug/' . $paddedI . '_' . $threads[$thread_key]['group'] . '.json', 'w');
fwrite($jsonFile, json_encode((array) $rawData, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonFile);
chown($jsonFile, 'www-data');
chgrp($jsonFile, 'www-data');
$update = true;
$threads[$thread_key]['synced'] = true;
echo($thread['synced'] . "\n\r");
}
}
if($update){
echo("saving updated syncing\n\r");
$jsonFile = fopen('xt.json', 'w');
fwrite($jsonFile, json_encode($threads, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonFile);
} else {
echo("no new files found for syncing\n\r");
}
}
?>

cheers

trw


Attachments: unnamed 
Subject: guess i will open for business soon
From: poster@anon.com (Anonymous)
Newsgroups: rocksolid.nodes
Organization: i2pn2 (i2pn.org)
Date: Tue, 26 May 2020 16:24 UTC
....and switch def2.i2p to vichan

just a little security research before, guess shouldn't take too long.

cheers

trw


Subject: opsec check is good
From: poster@anon.com (Anonymous)
Newsgroups: rocksolid.nodes
Organization: i2pn2 (i2pn.org)
Date: Tue, 26 May 2020 19:44 UTC
no bugs listed at bugtraq, none in my favorite hacker forum, none in google, no known hacks that I would heard about.
seems like we're goo to go.

cheers

trw


Subject: Re: basics are working
From: Retro Guy@rslight.i2p (Retro Guy)
Newsgroups: rocksolid.nodes
Organization: Rocksolid Light
Date: Tue, 26 May 2020 19:45 UTC
Anonymous wrote:

with some 11 exceptions my routine could glue all the threads together again.
this is still far from being elegant, mind you....

this code handles all the exporting from rslight to vichan.
-it checks the articles in the spool, -builds an index file and a watermark file (if necessary), -then resolves all the broken threads (fixed 10 iterations, have to make that dynamic later),
-then exports everything to vichan.

on the vichan side there is a modified post.php to get the messages.

so far the three parts (rslight, vichan and the script) are fully decoupled and individually called from cron.

That looks like a great way to handle it. So when complete, if I just add a $enable_vichan in config, then reference that in cron.php, I can just use that to decide whether to run the code at all. Is that correct?

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com



Pages:123456789
rocksolid light 0.6.5e
clearnet i2p tor