Operation AUTOADMIN(tm)

[Red Squirrel]

For those who might have noticed, I have not been doing much with the site lately, this is because it has come to a point where I have less time, as the time is spent working on repetitious admin related tasks such as news posting, backups, and occasionally making a script better or what not, but when something breaks it really costs allot of time and all the other stuff gets behind.

So I've started a new operation called Operation AUTOADMIN™ which is concentrated on making the site as automated as possible. The main things are:

1) News posting - About 90% of time
2) SQL backups - about 5% of the time
3) log processing - about 5% of the time

where time is the extra time that is put towards the site, and this time is only a few hours per day, now that I work.

So just an update on the progress:

The news posting parser is done, I'm working on the robot now and it's almost done. This involves mostly server side coding (my LAN server, not the Iceteks one) on the app that sits on the mail gateway. But it also involved some php code, and somehow it broke a script which I have yet to fix yet, it's really old nasty code, and I'll end up rewriting it. Since I got rid of the affiliate integration (The pull down menu so that affiliate news posts show their image and name from the DB)in the script it kind of messed the news poster.

The parser is a fairly large project, even when it's done, it will still require lot of tweaking but I hope to get it to parse about 80%-90% of news, and I may even ingore the news it can't post - it's up to webmasters to format their emails properly, some are hard to post by hand because of the bad format, so let alone trying to code a parser for it.




The log processing system is done, not automated, but is much better then my old one, and adding automation to it would not be too hard, but would require lot of error trapping in case of network failure, and may leave the site open to security issues as it would require a backdoor to be able to download the logs, but curently I'm satisfied with the system. The old one required me to download logs every day otherwise it got too big, not it makes seperate logs so I can actually hold off for over a month before touching it. The LAN system is also much better for log viewing.



SQL backups are the next big thing, I want to make it similar to the log system, where it's a one click download, rather then going through phpmyadmin to each DB.. get's long. (but yes, I do it often, so don't worry )


So really the main issue right now is the news system, because that's the time consuming one. Some hiccups here and there, but when you start seeing news posted by News-Bot, you'll know it's running smooth.

For those that post news, still check on that gmail account as right now the parser is only at about 50% efficiency so it still means about 10-20 items to post manually, and lot of them are not badly formatted emails where I say "too bad", it's just that the parser is not tweaked completely yet, I want to do the bot and make sure all that is smooth first.


For POST-operation activities, I have not decided yet but I want to make a few changes here and there to the forum, probably recode the html for the board index stat header on top, and make little template changes like that. Articles are really the big thing I need to get working on and my new job may help as I'm learning more about servers and stuff so it will give me subjects to write on, or give me ideas on completely different things.

I was even thinking of writing an article on the concept of automation, it would show general techniques on how to automate certain tasks, etc... but that's very vague, but the possibilities there are endless.






Archived topic from Iceteks, old topic ID:3358, old post ID:27178

[Red Squirrel]

Update:

The bot works great now, it was messing up big time earlier today, that's why not one single item got posted.

As for the parser, it's very iffy but we'll see how it goes for tomorow's news, hopefully the last few tweaks helped. The problem is that there's so many diferent email formats, it's very hard to make it universal. Lot of clients/people hit enter at all the wrong times, and don't hit it at the right time, making it even more confusing for the parser.

Archived topic from Iceteks, old topic ID:3358, old post ID:27181

[Red Squirrel]

Just an update on the parser. It's working a bit better, but now the major problem is that some emails aparently arn't making it in the database, even though they are passing through the parser, while some arn't even passing through the parser, but somehow are making it in the "parsed" box. So this is a serious issue I need to work out, since that's over half the emails. Some of the parsed emails have bad links though, but at this point, it's not my job to fix people's spelling/formatting mistakes in their news emails.

if someone does something stupid like

link: LINK
http://the.real.link/

then it's not my problem that the parser puts LINK as the link.

Archived topic from Iceteks, old topic ID:3358, old post ID:27222

[Red Squirrel]

OMG all this chaos was caused by a missing = sign. The program would do an abnormal termination when a parse would fail, so instead of continuing it would stop (obviously) so procmail would error out and just cancel that process and output the original email, so that's why it was not showing up as going through the parser. That said, the parser is still at a very low efficiency rate, like under 50, so still tons of tweaking to do. But I can only do so much without help of tech sites, if more tech sites can follow a decent format, it will make things easier.

Archived topic from Iceteks, old topic ID:3358, old post ID:27224

[Red Squirrel]

Another update: The parser is really starting to be tweaked and working well. I'd say about 50% efficiency now, and I'm adding a few tweaks here and there every day. By the end of the month I say it will be able to pull off 80% efficiency, or so. I may even make another bot that will post links only, as most parse failures are due to missing quote.

Archived topic from Iceteks, old topic ID:3358, old post ID:27301