So if your in the same boat and you have hundreds (thousands?) of RSS urls hard-coded into files across multiple servers I would recommend you get working, perhaps the following will be of use. I should mention at this point that Auction 2 Post users (and Bans, Wordbay etc.) will not suffer any fallout as they use the shopping api (which is sensible now, but back in 2006 wasn’t an accessible option.)

I have written a quick transformation script (js) which will take an old epn RSS url (http://rss.api.ebay.com/ws/rssapi?…) and transform it into a new one (http://rest.ebay.com/epn/v1/find/item.rss?…), if you only have a few urls to change over then it might be enough for you:

If you have a LOT of these urls in files (like I do) then you might want to use something a little more automated. In my case I wrote a complete FTP spider which flies for each and every file hunting for old ebay RSS urls, transforming them and updating the pages. This was fine in my case because mine were pretty standardly laid out, similar use etc. I wouldn’t recommend it if you use the extremity of the RSS parameters.

Anyhow as the FTP crawler was pretty huge of a code puke, I might release that separately at some point (it has other good uses ;)), here is the csharp for transforming old RSS urls into new ones though (Note: it doesnt deal with all the parameters, only the most common/ones I use regularly, it may need tailoring.)

static string transformEPNRSSUrl(string inputUrl)
{
    string transformed = "http://rest.ebay.com/epn/v1/find/item.rss?";

    //split url by ?
    string[] splitUrl = inputUrl.Split(new char[] { '?' });

    //split second by &
    string[] splitParams = splitUrl[1].Split(new char[] { '&' });

    //new params
    string newParams = "";

    //transform params
    foreach (string s in splitParams)
    {
        string retID = "";
        string retVal = "";
        string[] sV = s.Split(new char[] { '=' });

        //mostly :p
        retVal = sV[1];

        //see if tis required
        if (sV[0] == "sacat") { retID = "categoryId1"; }
        if (sV[0] == "satitle") { retID = "keyword"; }
        if (sV[0] == "afepn") { retID = "campaignid"; }
        if (sV[0] == "customid") { retID = "customid"; }
        if (sV[0] == "saprclo") { retID = "minPrice"; }
        if (sV[0] == "saprchi") { retID = "maxPrice"; }
	
        if (retID != "")
        {
            if (newParams != "") { newParams += "&"; } //for all but first
            newParams += retID + "=" + retVal;
        }

        //last few additions 24/06/2011
        if (sV[0] == "fts" && sV[1] == "2") { newParams += "descriptionSearch=true"; }
        if (sV[0] == "sascs" && sV[1] == "2") { newParams += "listingType1=AuctionWithBIN&listingType2=FixedPrice"; }

    }

    transformed += newParams;
    //following is some defaults for me, check these yourself, use:
    //http://woodylabs.com/scripts/ebay-epn-rss-url-converter.php
    transformed += "&sortOrder=EndTimeSoonest&programid=15&toolid=10039&listingType1=All&lgeo=1&feedType=rss";

    return transformed;
}

Use both the transformer script page and the csharp code at your own peril, it worked for me over hundreds of urls, mostly without a hitch, but thats not saying it will for you. Check your urls, after all each is probably worth £ to you. Visit the RSS urls and use the ebay tool validator to check the RSS feed has worked!

Just a quick mention of the positives of ebay deciding to rip out the old system (its not all a pain in the arse!):

• It reminded me how widespread my old ebay affiliate sites are, it was a good excuse to take stock and write FTPCrawler
• The whole system is much cleaner, parameter wise
• The introduction of multiple selection parameters is useful
• The new RSS generator is nice (as much as anyone uses the thing

The post Ebay Partner Network Changes RSS Urls (again) first appeared on Woody Hayday | Blog.

Website hacker entrance vectors (have any security holes?)

Common CMS, E-Commerce systems and forums (out of date or zero day, they all have or have had vulnerabilities) – WordPress, Drupal, OSCommerce, Gallery, PHPBB, VBulletin etc. etc. Particularly relevant here are the open source systems, but they are all susceptible – how many of these do you have installed where? For me these could of been answered with “a lot” and “some places”, clarity has now been restored but more on that later.

CMS Plugins (^^) – Often overlooked (especially by me), installed plugins can in themselves be entry vectors, often CMS’s push their communities to develop additional functionality for their system, which is a good thing, however if the system itself doesn’t deal particularly well with the security of folder structures or indeed how plugins are accessed they can offer ways in. Be careful with plugins which deal with file management and code execution (e.g. file attachers/uploaders etc.) Try to use late version highly rated plugins from reputable sources, with things like WordPress, plugins are low risk though as it has an excellent security model.

Bespoke server-side code and CMS’s – in my experience these is often LESS likely to get hacked, firstly “hackers” in this case are more likely to be script kiddies sitting in web cafes in some of the poorer world nations, they often use known exploits on common systems rather than trawl the web, searching for one off programmer mistakes. If you are behind the bespoke stuff leave out as many foot prints as possible and triple check everything. For bespoke stuff the most likely point of entry is simple SQL Injection, use SQL parameters.

FTP/WebDav – This really comes down to passwords as next indicated.

Passwords – Acquired by trojans or traffic sniffers, it becomes irrelevant what security you have in place across the whole setup if you don’t look after them properly. Avoid connecting to anything unencrypted (or at all if possible) on any network you don’t 100% trust, WIFI and wired, even if it’s a friends they could have a network sniffing Trojan on an idle machine. Install good anti-virus and protection software. Use Avast (free for private use) and Spybot Search and Destroy (these two are plenty.) Be careful with providing access to other users, whether it FTP, CMS, SSH, whatever – you may trust them but do you trust their computers?

Detection and Fixing – Realising you’ve been hit and fixing it

So after they gain entry, what would a hacker really do? Often with web hacking the motivation is kudos, money or sabotage – all are achieved through defacing, deleting or modifying web pages/logic and/or altering/downloading databases. A nightmare from the point of view of shared hosting users.

Depending on how they gain entry a hacker (or their automated executing code) may search through all of the files they can access, through ftp or server-side scripts, built to identify possible files to manipulate. They may download copies of things (e.g. databases!) but will likely set about cycling through all available webpage files and doing things such as:

• Inserting code within the page (iframes to bad websites, links to their websites – designed to improve their search engine rankings, redirect code which just shuttles people on to their sites)
• Replacing the file with a predesigned page (kudos fronts ‘this website was hacked by…’)
• Replacing common server-side functions and variables (e.g. replacing all the send values on email scripts to forward emails to an account other than the owners)

If you have a single site, or 10 or something the chances are you personally check them all every now and again, getting a little facetime at least once a week say. In this case your opportunity for hosting a hacked site is 7 days, not awful, not great – Google and browsers will start blocking your site if the hacker has inserted any code going to malware or similar, and otherwise may start to drop you down the rankings if your site now displays a ‘hacked by..’ page instead of your wholesome site.

In the case of a lot more sites on the host this can mean no detection for a longer time, if no system is in place, often first recognised through a drop in statistics/earnings (more likely earnings as in the case of iframed malware a change in the number of hits can be not hugely obvious.)

To add confusion to the mix it’s not unknown for hackers to mask their changes to you, it’s very easy with .htaccess files and php/asp headers (for example) to show content relative to its viewer. E.g. the hacker could shuttle people coming in from Google to a hacked page but people that access the site directly (typing it in) get shown the normal site. Furthermore they may not hit every site you have, perhaps a handful of random choices, some folders not others, a smokescreen like attack which could change each time.

Chances are once you get all your pages fixed and get around to looking at where the security hole is that when you recheck your sites they would have been hit again. This tells you two things, 1. The hacker is relentless (or more likely has a relentless automated program, exploiting 24/7) and 2. You have not plugged the security hole. Or if you are really unlucky you are being hacker tag teamed.

So anyway, detection. How do you go about knowing the integrity of your web portfolio? What if it spans 10 shared hosting accounts or 4 servers? Well likely if you own your own server you have spent the time/cash in locking everything down, what I suggest here would be useful to you guys but you may already have a better solution in place.

Currently there are a bunch of services which will do this for you, of which I have tried zero. “Monitoring” services are available worldwide ranging in prices drastically, for me though even the high end services didn’t offer a full set of features and were mostly hugely overpriced but for the top 50% of the portfolio, not effective for me.

The good things about using external monitoring services are obvious but none seemed to be able to offer realistic change monitoring (e.g. WordPress blogs may change content between <div id=”whatever”> and </div> every hour but the rest of the page should stay almost the same.) It is important they see the addition of malicious code to good pages and not throw constant false alarms. For ‘this website was hacked by..’ pages though they probably do a good job (as well as malware detection.) Uptime monitoring is also common as part of the packages, useful without doubt.

I suggest another way though of monitoring an established portfolio, that is the way I have resolved my recent hacker attacks, a realistic option for shared host/anywhere-in-the-world-with-a-laptop client based use. Ultimately an extension of a few older applications I wrote to manage a growing portfolio, weathered by several hacks across accounts within the past 6 months – Hard checks of every important file.

At first I wrote my system to simply allow me to take stock of the sites and CMS’s I have in place, to work out possible security holes from behind the scenes – but it turns out it works surprisingly well in identifying breaches. By checking your actual live file structure (.htaccess, index.php, default.aspx, index.html etc.) against a known correct file structure snapshot, you take the whole http part of the checking out of the loop, effectively making it a higher level integrity check than external services can ever offer.

By making Checksums of every critical file (often hackers just hit index.php, index.html, default.aspx etc.) within a given ftp/file structure and then automatically rechecking at scheduled points it becomes easy to minimise your window for financial fallout from hackers. This may seem like a time/bandwidth/processor consuming task but in actuality 100 websites with WordPress installed could be checked in a few Mb of download – in terms of modern data use that’s a few browses of a facebook photo gallery. What’s more it can run in the background, only prompting you on changes to files, as frequently as you want.

The side benefit of producing complete hosting account checksum snapshots is you are also able to accurately backup a working copy of your hosting account. Built into the checking process this means that you can then correct hackers’ malicious changes with a click of a button.

This of course does not take into account more hard-file based websites, database changes or regularly altered sites. I recommend automated screenshots to cover these or the combination of external monitoring services and integrity checking.

I have written an alpha release of this system (named Website Integrity Checker for now) and will gladly distribute/discuss it if you drop me a comment below. A beta copy might make its way out sometime.

The post Hackers wrecking your shared host account? Check your Website Portfolio Integrity first appeared on Woody Hayday | Blog.

This time however, with a new push for near as possible to 100% outsourcing I have found a good writer on ODesk, along with some supporting staff to deal with some other *secret* processes – so far so good – I took the time to interview, had one issue out of 6 or so employees but I guess I could of put more time in to harder interviews, in this case the cost was negligable.

The best bit? While away exploring Denmark I was more effective than any normal 10 days, plus I had a great time. Win. Next up I am going to find somewhere to live, probably in London – With a bit of luck this new batch of sites will help

Question is – how big of a human/bot machine can you build using Amazon Mechanical Turk, Odesk Employees, csharp bots (sometimes actually providing orders..lol) and the web at large?

The post ODesk and outsourcing first appeared on Woody Hayday | Blog.

2010 Has been a good year for travel, from the sandy cocktail beach’s and fast paced mountain biking through lychee groves of Thailand across to the lake of Pokhara and the remote Bandipur in Nepal; sidestepping riots in both countries and a ridiculous ash cloud, through grace or luck. From central park and the Guggenheim to the Robert Moses beach and Long Island, New York was epic; and now in 48 hours or so on to Agadir, Marakesh, Casablanca and Fez, Morocco, its been a good year for travel.

While at home I have also picked up climbing, something I wish I had done more of while in Thailand. The Castle near Finsbury park is a fantastic location, especially when visited with a few good friends. I need to do more of that. I also look forward to a time when I have a piano in a house of mine so I can truely take that forward and carry on learning

Through work and exploration this year I have absorbed C#, something I am quite happy with as it bridges the gap in my skillset between open-source and MSFT, between web programming and local app development. In my opinion it is also a good year for Microsoft, despite their mediocre year on NASDAQ; they are releasing much more rounded products and seem to be beginning to see the benefits of what I see as a change in strategy. Either way C# has lead me to refine my rapid application development and brought the brute modern processing power directly into use, I would thoroughly suggest it to supplement your language base if you are leant heavily towards php/LAMP (while I would only use it over php for few of my future web projects.) RAD in C# let me experiment with an arbitrage betting program and write a quick Reddit wallpaper grabber (yes I know it needs fixing); as well as a bunch of other stuff like a Filezilla bulk user import, an SQL->php hardcoded rewrite of a huge site that meant 300% quicker loads, a remote image processing app, tons of little sharepoint integration bits, a full sharepoint 2010 site migration routine and also a true 3rd gen website asset management tool, which I will perhaps post more about.

Near U has had a fairly good year and I have added a few more sites to the portfolio (I still need to diversify though) – mostly bolstering and fulfilling old ideas for domains that have been stagnating, I hope to shift this up a gear to bring it all back into line now I have the freedom to do so; the cold financial climate does seem to have slowed people buying cars, specifically though on-line auctions this past quarter but I have high hopes for the new year. Part of managing this portfolio of now more than 200 sites has been writing up a proper asset management tool geared towards websites (mostly affiliate) as assets of value, csharp has allowed me to write this multi threaded, flexible, detail rich program in a ridiculous short stint of evening programming, perhaps it may even make its way as a sell-able tool, this however is to be seen.

In the last few months I have also been running with Sharepoint 2010 (another example of microsoft getting themselves together) – which although on its preface is limited or “nothing new” in fact has quite sufficient enough depth to hold weight as an intranet solution. I doubt you will find better workflow, versioning, integration and scope in a web-based solution (even if it is £120k for enterprise for 1000+ users) – developers should not be scared of it. To supplement a sharepoint installation I would also advise the familiarisation of developers with WCF, microsofts best answer to web services moving forward.

There has been a few good books recently which have also left a resounding impression, The Road Less Travelled (by M. Scott Peck) left me thinking more about the sub concious and revealed new angles on understanding people will that I will forever remember. In Nepal I left my copy of The Art of Happiness on a bus, but luckily found another copy in a bookshop in Pokhara; it is a fantastic book that taught me more than anything to see things more clearly and seek contentment truthfully. The Richest Man in Babylon was suggested to me by a friend who is investing in silver, which seems sensible – what a fantastic little book! some basic rules on managing money are told through a Babylonian narrative which is fun and quick to read. If reading like this was more present in our schools we would all be wealthier, at least everyone except producers of instant gratification goods. I also subscribed to Harpers this year, after finding it in Bangkok airport with hours to kill. It has significantly inspired me to proceed with writing my novel, it is always a great read and the harpers index is good:

Percentage of U.S. car owners who keep maps in their glove compartments: 50, Sunglasses: 23, Gloves: 0

Estimated value of Chinese household income that goes unreported: $1,400,000,000,000; Portion of China’s GDP this represents: 1/3

Number of poisonous dead mice the USDA airdropped into Guam this year to eradicate an invasive snake species: 316

Number of times between January and June that google turned over user information to government investigators: 4,287

Might have already posted about this before but Evernote has served me well all year, allowing me to dump my ideas, wacky thoughts or notes into a single searchable repository, wherever I am. I also started backing up to the cloud (cant wait to write new software that takes advantage of the amazon storage cloud) via JungleDisk, which seems to be pretty spot on for me, and backing up iphone contacts to Google contacts was also a helpful find. If you guys are not yet using Chromium (NOT Chrome), that’s another no-brainer. Install it, you’ll see what I mean (sorry Mozilla, you were trumped on speed.) Spotify has continued to be epic, even if the lack of nichey remix’s and the removal of some epic songs is irritating. Finally I have to say that Microsoft dropping their Express versions of visual studio 2010 and SQL Server is both useful, and strategically a good idea. Their IDE is pretty ridiculously good, and providing free (although slightly limited – SQL max db of 5gb for example) versions will bring in the hordes, and if they develop anything decent, chances are they will shell out for licences legitimately.

Best music find of recent months: Emancipator

Looking forwards I now cant wait to finish my novel, write a bunch of new useful softwares and in 2011 start another business, perhaps a software house selling software as a service? I have a pretty devilishly good idea that will leverage the amazon clouds scalability. Perhaps. Either way I fly to Morocco in 48 hours and I haven’t even thought about it yet. Better go.

The post Forget Inertia – A new era. first appeared on Woody Hayday | Blog.

So I happened across this post on foreign exchange arbitrage and got distracted for 4 hours into working out if there were any opportunities for using arbitrage betting from here in the uk, what with online gambling being so big and tennis being a simple outcome game I thought I would start with the tennis and see where it took me. More an excuse to practice my rapid application dev than anything else.

If you skip to the end of this post you can see the outcome, or read on, I will be brief.

So I started in excel, found a nice summarising scraper site which gives you all the odds from different bookies in one place (how convenient – I bet they make good affiliate revenue) and did the maths. Within 4 games that evening I found one which worked. If I had placed bet’s across 2 bookmakers totalling £1000 at that moment I would have made a profit of £10.74 (1.74%), guaranteed (unless some natural disaster killed both tennis players?!?)

Great. Well its not, but 1.74% was a start that could be turned into a healthy profit, provided easy money in, easy money out, repetition and scalability (not that I expected that but I thought I would go with it.)  The math I was using was as follows:

Where x was odds on outcome1, y the odds on outcome2
if c was less than one then it would be an arbitrage bet which held profit (this is a simplified version of the math, which I am fairly sure is correct)

This gave me a percentage of return based on a formula, next I wrote a quick scraper which pulled match’s from the odds aggregater, it pulls all tennis games across different championships and arena’s. I wrote the scraper in csharp, a variant on a rules based spider-scraper system I wrote in php a few years ago for ripping various things off the net into db’s. So 1 hour or so later at the flick of a switch I now had all tennis match’s popularly available to gamble online on in the uk with odds, bookie details etc etc.  I love RAD in c#. Apply the math to this and you get the following:

Yeah, its ugly, but it works.

So yeah perhaps that early game I found that made nearly 2% was a fluke or a miscalculation, perhaps arbitrage betting is as fluidly fixed by the free market as forex is, either way in the few days I have checked I haven’t found any arbitrage bets that are viable at all, let alone any healthy returns. There probably is scope in the practice, perhaps with 24 hour rate watching you might catch the book-keeper fluctuations and make the odd 1-4%, but really, for all the hassle of multiple accounts and bet-shop money holding its probably not worth it. Good excuse to re-write my scrapers in csharp though, and do a little maths.

*Gambling is bad, mmmmmkay? Its also worth noting that while if you couple finding this low % profit with the £10 free bet crap that all of these online bookies offer it would seem like a great idea, or at least a good one. Its probably not. These business’s run a filter model for your cash, good marketing, low number page hops and quick card transactions makes it easy to filter in cash, but isn’t so easy to get out returns, also quite often the free initial bet or bonus comes with proviso’s which wreck the rate of returns. Its no surprise, its probably easier to earn your money elsewhere, perhaps by starting up as a bookie? :p

The post Arbitrage Betting – Programmatically finding arbitrage bets first appeared on Woody Hayday | Blog.

May was a month of brick lane curries, a new art exhibition (silent city), mini golf and a bunch of other blurry stuff.

June was a good month (on the whole apart from hackers and google caffeine?!?), did incalculable amounts of coding, new site rollouts, tennis, monopoly, booked new york flights and some other stuff. Wrote a lot, drank a lot of wine, updated a bunch of auction2post sites because eBay updated their api (for the better) – standard summer month.

I started 4 draft posts across these two months but none seemed to stick, it’s not like the months were dry, posts boning csharp just didn’t fit, I was going to post csharp iis and multisql management code, a nice project I wrote to datamine from ebays new api and I was thinking of digging out my seo hub php based hub and rewriting it as a multi threaded windows app, but yeah sometimes stuff just doesn’t seem to hold enough value.

As ever though projects continue to crop up and evolve. I am working on a stupidly simple file system snapshotter after an incident with a second hacker, I will probably post about ripping data from the Nike app on the iPhone, if it isn’t up already somewhere else. Also I will make a quick program to save the top 20 wallpapers from reddit to a folder so windows 7 can automatically show me new epics (theres even an rss!) if know one else saves me the effort. Unless that is, all this stuff gets superseded. Mostly it’s just me biding my time and making broader plans

Look out for possible random posts on business intelligence, the stock market and brain architecture too, as they seem to keep cropping up.

The post Unpublished May and June 2010 first appeared on Woody Hayday | Blog.

PHP is the tool of the open source project. (There are of course companies that use it (my companies do) and huge online websites/behemoths of information that use it as a server side language.) but overall the corporate world, any web software with intent to move to larger markets and bridge the gap to desktop apps codes in ASP and now ASP.NET. There was nothing I couldn’t do in PHP/mySQL that I wanted to do, it for me was a very good toolset and fulfilled my needs as far as I could see them. But it would lack if you wanted to move towards desktop applications, and its lack of association to Microsoft does act as a restriction in some ways. So there it is ASP / ASP.Net needs to picked up. For me this was more a conversion than picking up a new skill, I took PHP and hammered that knowledge into ASP Syntax – this actually works nearly entirely for the most part, you get over the differences very easily if you have done even a bit of visual basic before (as ASP in its default form is essentially vb – C# is also excellent.)

I just took a course in “ASP.NET Scalable web applications using AJAX” (Learning Tree in euston – would recommend it – taught by an excellent asp.net consultant, Richard Howells) which affirmed a lot of programmatic choices I have previously made and enlightened me to improved structures for scalability. The thing I most took out of it though is how much work microsoft have put into their IDE (Visual studio.) VS2008 is pretty phenomenal if you come from hand coding everything yourself. I can see how developers get wooed by intellisense and ease of access, they intend to make it all easy – every functionality provided by web technologies in their control based environment. For me it still remains though that in providing a huge framework of simplicity to every user you do carry a certain amount of redundancy, that is while it may take longer to code pure PHP \ Javascript it will still do specifically what you want it too, and only that. Potentially with the microsoft IDE you can create what you want entirely and then cut the fat so to speak afterwards – there are substantial speed benefits with their project based management, this methodology however is not my first choice.

For any person wanting to seek employment or understanding in web development I would highly recommend jumping strait into PHP or ASP.NET (probably PHP unless you have a requirement for ASP – ASP.NET C# Pays better than PHP here in the UK.) After learning the basics of HTML, Javascript and CSS, PHP or ASP brings you clearly up another step. I am available for code/developer mentoring/support as of Summer 2009.

The post Developers Arsenal PHP to ASP Jump first appeared on Woody Hayday | Blog.