for some reason the almost identical code sequence that works for send.php in rslight for posting messages to a news server (using the post command),
produces a 480 - posting not allowed when i try to use it for vichan (with the same server). guess i do something for the auth.
this is clearly not RESTful :-)

--
Posted on def2

Anonymous wrote:

for some reason the almost identical code sequence that works for send.php in rslight for posting messages to a news server (using the post command), produces a 480 - posting not allowed when i try to use it for vichan (with the same server). guess i do something for the auth.

Are you setting $remote_auth_user and $remote_auth_pass in the file? In rslight these are pulled from config. If that's ok, when exactly do you see the error? After AUTHINFO commands? After POST command?

The function nntp2_open in send.php uses language variables to display some errors. If you're not linked to rslight, you probably won't see the errors from it. You can just change those lines to display the text you want for the error.

BTW, the function nntp2_open in send.php and the modified vichan_nntp.php are identical functions. If you link the two files (include one in the other), you'll need to drop the function from the secondary function.

send.php does not include some functions it uses that are in newsportal.php. If you are running it not including that file, it probably won't work. The best way to handle it for now is to just copy the functions into the send.php you want to use. (vichan_send.php maybe?)

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

when exactly do you see the error? After AUTHINFO commands? After POST command?

i am not sure, because with telnet everything works, and with the program, i don't see exactly what happens. some screwing up the auth, i think.
actually, ihave or takethis would be better than post.
this will take a but more tinkering.

--
Posted on def2

Are you setting $remote_auth_user and
$remote_auth_pass in the file?

i set the vars right (passing them on from the calling script), tested that before.

when exactly do you see the error? After AUTHINFO commands? After POST command?

i don't know, because i don't see the intermediate stages. i guess i also had a mistake in parsing the the responses from the news server.
doing the same in telnet i get no error at all. guess it is not exactly the same that i do in telnet.

The best way to handle it for now is to just copy the
functions into the send.php you want to use.
(vichan_send.php maybe?)

that is the question. i tried to make nntpchan.php from vichan working, which it did not for some reason. when i do the psoting to the news server there, i can't (or don't want) to drag a lot of libs in, because that will end up messy i guess, and i don't understand the vichan code well enough to really integrate it well.
the other option is to do the posting in the import script i have already (which would also become an export script than), in this case one more lib doesn't hurt. or maybe the posting function could be included in vichan_nntp.php ?
what i don't understand yet is how rslight can use POST to post an article with an existing message id ? i understand that normally the news server generates the number for you when you use POST (like for fudforum for example).
forwarding existing messages should (according to RFC) be done with IHAVE or TAKETHIS.
not that i will complain about this point if i can find a way to make this work, mind you :-).
the advantage of using POST is maybe that more news servers will let you use it vs streaming with IHAVE or TAKETHIS.
but anyway, it is maybe best to take that code outside from vichan and just continue to drop the messages as files in a defined directory. this way i can also implement a followup (post successful or not).

cheers

trw

--
Posted on def2

Anonymous wrote:

The best way to handle it for now is to just copy the
functions into the send.php you want to use.
(vichan_send.php maybe?)

that is the question. i tried to make nntpchan.php from vichan working, which it did not for some reason. when i do the psoting to the news server there, i can't (or don't want) to drag a lot of libs in, because that will end up messy i guess, and i don't understand the vichan code well enough to really integrate it well.
the other option is to do the posting in the import script i have already (which would also become an export script than), in this case one more lib doesn't hurt. or maybe the posting function could be included in vichan_nntp.php ?

That shouldn't be a problem, especially since send and retrieving articles share some functions.

what i don't understand yet is how rslight can use POST to post an article with an existing message id ? i understand that normally the news server generates the number for you when you use POST (like for fudforum for example). forwarding existing messages should (according to RFC) be done with IHAVE or TAKETHIS.

inn and rslight news servers will accept articles which include a message-id, and retain that when receiving via the POST command. Both servers will create a m-id if one is not included, but will not change it if it is included.

Consider that rslight spools a message posted to it right away. It can't wait until after it's posted to an upstream server to then get a m-id, it needs one immediately to spool a proper article. It is intentional that when clicking 'Post' when sending a message in rslight, you need not wait for communication with an upstream server before moving along. On i2p especially, this is important.

But when posting to inn or rslight from vichan, including a message-id in your message is not required, it's just supported.

not that i will complain about this point if i can find a way to make this work, mind you :-). the advantage of using POST is maybe that more news servers will let you use it vs streaming with IHAVE or TAKETHIS.

Exactly. Most servers will not allow IHAVE or TAKETHIS (streaming), but any server you can post to with a reader will respond to POST properly.

but anyway, it is maybe best to take that code outside from vichan and just continue to drop the messages as files in a defined directory. this way i can also implement a followup (post successful or not).

Since you already have a properly formatted message coming out of vichan, it really should be a simple matter of (taken from send.php):

  $messages = scandir($outgoing_dir);
  foreach($messages as $message) {
    if(!is_file($outgoing_dir.$message)) {
      continue;
    }
    echo "Sending: ".$outgoing_dir.$message."rn";
    fputs($ns, "MODE READERrn");
    $response = line_read($ns);
    if (strcmp(substr($response,0,3),"200") != 0) {
        file_put_contents($logfile, "n".format_log_date()." ".$config_name." Unexpected response to MODE command: ".$response, FILE_APPEND);
      return $response;
    }
    fputs($ns, "POSTrn");
    $response = line_read($ns);
    if (strcmp(substr($response,0,3),"340") != 0) {
      file_put_contents($logfile, "n".format_log_date()." ".$config_name." Unexpected response to POST command: ".$response, FILE_APPEND);
      return $response;
    }
    $message_fp = fopen($outgoing_dir.$message, "rb");
    while (($msgline = fgets($message_fp, 4096)) !== false) {
      fputs($ns, $msgline);
    }
    fputs($ns, ".rn");
    fclose($message_fp);
    $response = line_read($ns);
    if (strcmp(substr($response,0,3),"240") == 0) {
      unlink($outgoing_dir.$message);
      file_put_contents($logfile, "n".format_log_date()." ".$config_name." Posted: ".$message.": ".$response, FILE_APPEND);
    } else {
      file_put_contents($logfile, "n".format_log_date()." ".$config_name." Failed to POST: ".$message.": ".$response, FILE_APPEND);
      continue;
    }
  }
  return "Messages sentrn";

The above is sending from a file, but could just as easily be sending from a variable.

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

thanks for the code. i had to change a few places, like:
echo "Sending: ".$outgoing_dir.$message."rn";
to
echo "Sending: ".$outgoing_dir.$message."\n\r";
and, more importantly:
fputs($ns, "MODE READERrn");
to
fputs($ns, "MODE READER\r\n");
and like that in all the places sending this sequence.

in the process i discovered that subjects with "\" will break something in bbs :-).

anyway, it is working, and vichan posts directly to news servers now.
so now, it can run without any local news server, meaning you should be able to sync any existing installation to newsgroups.
very cool.
i need to clean up my code, it is still horrible right now, and i guess it can be done in half of the lines.
after that, maybe we should publish that somewhere it can be found (thinking some fork on github or so...). i know i said earlier a fork would be too heavy for me, but maybe we can offer it to some existing fork ? what do you think ?
i just have the feeling it would be unfair to build on something existing and than not sharing your results.

anyway, it is working. very nice. :-)

cheers

trw

--
Posted on def2

btw, this setup should also be a compliant implementation of overchan. i wonder whatever became of this project. it was really huge in terms of numbers of nodes once.

--
Posted on def2

Anonymous wrote:

thanks for the code. i had to change a few places, like:
echo "Sending: ".$outgoing_dir.$message."rn";
to
echo "Sending: ".$outgoing_dir.$message."nr";
and, more importantly:
fputs($ns, "MODE READERrn");
to
fputs($ns, "MODE READERrn");
and like that in all the places sending this sequence.

Yes, rslight removes the backslash when displaying the code. What I sent had the backslashes but they were stripped for display.

in the process i discovered that subjects with "" will break something in bbs :-).

There's always something new in everybody's code :)

anyway, it is working, and vichan posts directly to news servers now. so now, it can run without any local news server, meaning you should be able to sync any existing installation to newsgroups.
very cool.

One of the links I sent in r.s.encryption is an rslight nntp server. If you want to test syncing vichan directly to rslight and back, it might be a great test to prove no inn servers are required. I already sync one rslight site to another successfully. (My freebsd test site to rslight.i2p)

i need to clean up my code, it is still horrible right now, and i guess it can be done in half of the lines.

Uh, yeah, me too :)

after that, maybe we should publish that somewhere it can be found (thinking some fork on github or so...). i know i said earlier a fork would be too heavy for me, but maybe we can offer it to some existing fork ? what do you think ?
i just have the feeling it would be unfair to build on something existing and than not sharing your results.

I agree that when it's working well, provide a download at least. That's all I do for rslight. I have no github, cvs or anything, but it can be downloaded. I'd do the same for vichan. If someone wants to fork what you wrote, that's great, but it's nice to keep your own fork for your own mods, etc.

anyway, it is working. very nice. :-)

I'm amazed at how fast you got this working! Great work!

Retro Guy


--
Posted on: Rocksolid Light
www.novabbs.com

Retro Guy wrote:

Anonymous wrote:

in the process i discovered that subjects with "" will break something in bbs :-).

There's always something new in everybody's code :)

I noticed earlier today that rslight's overboard did not display koi-8 correctly. The article would display properly, but not in the overboard. That's fixed now :)

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

Anonymous wrote:

btw, this setup should also be a compliant implementation of overchan. i wonder whatever became of this project. it was really huge in terms of numbers of nodes once.

It's odd how it just kind of went away. It seemed to work fine.

--
Posted on: Rocksolid Light
www.novabbs.com

it might be a great test to prove no inn servers are required.

did that with my own instance already, it is working.

--
Posted on def2

anon <anon@anon.com> wrote:

POST /post.php HTTP/1.1

In case, you can see JSON from my read-only newsreader here:

https://nnrp.alphanet.ch/newsgroup/rocksolid.programming/%3C8cb53de068f68ce971aa235b44272aab%241%40news.novabbs.com%3E?mode=json

Implementation:

   if ($self->param('mode') eq 'json') {
      my %headers;
  
      my $headers = $a->headers;
      # yeah, not simple re
      #    - to make the loop work, we need to not remove the final \n if any, so we use
      #      a look-ahead: https://stackoverflow.com/questions/3926451/how-to-match-but-not-capture-part-of-a-regex
      #    - we accept continuation lines
      #    - maybe the header re is a bit too strict
      #    - 'rest' should be empty if all was right
      $headers =~ s/(^|\n)([a-zA-Z0-9\-]+):\s?(.*)(\n\s+.*|)(?=\n|$)/$headers{$2} = $3 . $4; ""/eg;
  
      $self->render(json => { 'headers' => \%headers,
                              'body' => $a->body,
                              'headers_raw' => $a->headers,
                              'rest' => $headers
      });
   }

do you want to start soon, or wait a little longer ?
if you want to start soon, i would also start to work on the direct syncing (so: pushing messages from one instance of vichan to another via http).
if not, i will focus on something else first (like: dealing with attachments in a good way).
and talking about attachments: can we agree on limiting the number of attachments to one for the rocksolid groups ?

cheers

trw

--
Posted on def2

In case, you can see JSON from my read-only newsreader here:

yes i can see json :-)
what about it ? what do you achieve with that code ?

--
Posted on def2

Anonymous <poster@anon.com> wrote:

yes i can see json :-)
what about it ? what do you achieve with that code ?

I saw some discussions about people wanting some JSON access to rocksolid, so
this is another.  As it was trivial to add JSON support to my read-only
web NNRP interface, I did it.

I saw some discussions about people wanting some
JSON access to rocksolid, so

I guess you refer to Retro Guy and myself discussing. The interface I introduced on one of my nodes is using json now to receive posts that are synced from other nodes (+ there is a json api already that comes with it).

 As it was trivial to add JSON support to my read
only web NNRP interface, I did it.

yeah, i took a look at the link (also linked your site here, btw).

cheers

trw

--
Posted on def2

-fix date for message import

this one turns out to be really tough. i think it was not meant to work that way (so, to have the original posting time).

i have this line here in the function post (functions.php), which sets the time for the article to be inserted:

$query->bindValue(':time', isset($post['time']) ? $post['time'] : time(), PDO::PARAM_INT);

so, if we don't have a time, we take the current time. this is how i read this line.
but for some reason even if the variable is set to the original time of the article, the current time is taken instead.
maybe it has to do the complicated callback construct that they use (which i don't fully understand).

:-(

trw

--
Posted on def2

had nothing to do with the callback function, i was simply setting the variable after it was used.

new status:

open:
-fix proper handling of attachments from imported messages
-code simplification and cleanup

closed:
-fix date for message import
-fix catalog link in the footer, catalog is off: i thought i had to change the templates, but it is just a switch in the config
-fix threading: fixed some issues in the import script
-mime encode subject if needed: thx to Retro Guys line

after i have fixed the part with the attachments, i will reimport all the messages one last time.

cheers

trw

--
Posted on def2

open:
-code simplification and cleanup

closed:
-fix proper handling of attachments from imported messages
-fix date for message import
-fix catalog link in the footer, catalog is off: i thought i had to change the templates, but it is just a switch in the config
-fix threading: fixed some issues in the import script
-mime encode subject if needed: thx to Retro Guys line

Retru, do you want to test this as well ? Do you want to roll out vichan, or wait a little longer (until all the bugs are hope fully squashed) ?

cheers

trw

--
Posted on def2

trw wrote:

open:
-code simplification and cleanup

closed:
-fix proper handling of attachments from imported messages
-fix date for message import
-fix catalog link in the footer, catalog is off: i thought i had to change the templates, but it is just a switch in the config
-fix threading: fixed some issues in the import script
-mime encode subject if needed: thx to Retro Guys line

It's really looking good! I haven't checked out imported messages with attachments yet, but it sounds like you have it working :)

Retru, do you want to test this as well ? Do you want to roll out vichan, or wait a little longer (until all the bugs are hope fully squashed) ?

I would like to check it out, and eventually set one up as a node. I'm interested in seeing the nntp code (send/receive) and maybe also find some nice ideas for future stuff.

If it can be placed in a tarball or something, I'll try it out. I think it requires mysql, if I'm not mistaken.

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

It's really looking good! I haven't checked out imported messages with attachments yet, but it sounds like you have it working :)

yes, it looks like it (you can check on def2.i2p/ts how it displays). of course all depends on getting the attachment first (so the rslight code). thats's why it's great news for me that you solved these darn inline attachments (like def3 makes).

I would like to check it out, and eventually set one up as a node.

ok, i will upload an archive with the code (attached to this message).

I'm interested in seeing the nntp code (send/receive) and maybe also find some nice ideas for future stuff.

well, you will see a lot of your own code :-).

the question for me is how soon you will roll out. if it is sooner, i will work on one last point first before the cleanup, which is the direct syncing between two vichan nodes, using the modified post.php. if you will wait a little longer, i will clean up the code first and than add that feature when needed.

If it can be placed in a tarball or something, I'll >try it out. I think it requires mysql, if I'm not >mistaken.

yes, you need mysql and pdo to run vichan. if you go back to the original tinyIB code, there is an option to run with flat files. but the setup process of vichan was absolutely painless for me, so i did not really investigate further.

let me know if you need help to run the code, it is still very rough.
you will need to adapt sync.php and nntpchan.php to reflect your directories and news server settings.

cheers

trw


Attachments: unnamed (application/zip)

please find attached the latest code archive, the other one was missing a file. also the update concerning the attachments is included.

cheers

trw


Attachments: vichan_sync_v0.11.zip (application/zip)

Anonymous wrote:

please find attached the latest code archive, the other one was missing a file. also the update concerning the attachments is included.

Thanks! I'll take a look at this in a couple of days. I'm quite curious to see how vichan is written.

Retro Guy


--
Posted on: Rocksolid Light
www.novabbs.com

Retro Guy wrote:

Anonymous wrote:

please find attached the latest code archive, the other one was missing a file. also the update concerning the attachments is included.

Thanks! I'll take a look at this in a couple of days. I'm quite curious to see how vichan is written.

I've downloaded and installed (on top of vichan-devel). I kept the dir structure from your zip and put it into the devel structure. Then installed (seems vichan requires php-mcrypt) but that's not listed in Requirements in the README).

Anyway, it installs easily, and I set the nntp server in config file. I then found sync.php and ran it as www-data user in the directory it's in (./debug). Not sure if I was supposed to run it, or move it elsewhere, or run it from another dir, but I ran it from there. It created the thread files in ./threads fine (nice).

What I don't know is how to I tie those to the boards in vichan? So, what did I do wrong, or what have I not done yet that I need to do? :)

Thanks,

Retro Guy




--
Posted on: Rocksolid Light
www.novabbs.com

Then installed (seems vichan requires php-mcrypt) but that's not listed in Requirements in the README).

Strange, either I had that installed already, or there is something different between our installations. I don't recall having to do that.

Anyway, it installs easily, and I set the nntp server in config file.

Good, that was the same for me, installation was easy.

I then found sync.php and ran it as www-data user in the directory it's in (./debug). Not sure if I was supposed to run it, or move it elsewhere, or run it from another dir, but I ran it from there.

That's fine. It can run from anywhere, and you can run it from cli or call it from cron.
Did you put gpost.php in the webroot (next to post.php) ?

It created the thread files in ./threads fine (nice).

There should also be a dir named "watermarks". If not, something went wrong.

What I don't know is how to I tie those to the boards in vichan?

Each board needs to be set in its config file, and there you make the connection to the newsgroup.
Also in sync.php there is an array called "translation" which needs to be set (I will take that from the central config later).

So, what did I do wrong, or what have I not done yet that I need to do? :)

I would suggest to start with one group only:
-set it in the central config
-set it in the board config
-set it in sync.php
and let run again.

What is the content of error.log and application.log ?

cheers

trw

Posted on def4