Subscribe: Rasmus' Toys Page
Added By: Feedage Forager Feedage Grade B rated
Language: English
base  continue reading  ddr  debian  event  gearman  libdrizzle  new  nifty trick  oauth  queue  request  sata  server  xhp  zeromq 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Rasmus' Toys Page

Rasmus' Toys Page


megasync for Debian 9 Stretch

(image) Like most of my posts here, this is mostly a note to myself so I don't forget how I did it.

I Moved to Debian 9 on my desktop box at home and everything works great except I occasionally use and they don't provide a Debian 9 build. It would be great if they just provided a statically linked generic Linux binary, but they don't. So, to make it work, grab their Debian 8 .deb file.

Continue reading "megasync for Debian 9 Stretch"

Upgrading PHP on the EdgeRouter Lite
(image) After nearly 7 years of service I retired my Asus RT-16 router, which wasn't really a router, but a re-purposed wifi access point running AdvancedTomato. In its place I got a Ubiquiti EdgeRouter Lite. It is Debian-based and has a dual-core 500MHz 64-Bit MIPS CPU (Cavium Octeon+), 512M of ram and a 4G removable onboard USB stick for < $100. The router is completely open and, in fact, any advanced configuration has to be done from the command line. The Web UI has been improving, but there are still many things you can't do in it. In other words, exactly the type of device I prefer.

Continue reading "Upgrading PHP on the EdgeRouter Lite"

Building a NAS
The HTPC box and various computers around the house use a mix of internal drives, external USB and eSATA drives. It is quite a mess, and backups are sporadic at best. The HTPC especially has grown organically with USB and eSATA as it needed more and more space.

(image) So it was finally time for a decent NAS.
Continue reading "Building a NAS"

ZeroMQ + libevent in PHP
While waiting for a connection in Frankfurt I had a quick look at what it would take to make ZeroMQ and libevent co-exist in PHP and it was actually quite easy. Well, easy after Mikko Koppanen added a way to get the underlying socket fd from the ZeroMQ PHP extension. To get this working, install the PHP ZeroMQ extension and the PHP libevent extension. First, a little event-driven server that listens on loopback port 5555 and waits for 10 messages and then exits.


getsockopt (ZMQ::SOCKOPT_EVENTS) & ZMQ::POLL_IN) {
        echo "Got incoming data" . PHP_EOL;
        var_dump ($arg[0]->recv());
        $arg[0]->send("Got msg $msgs");
	if($msgs++ >= 10) event_base_loopexit($arg[1]);

// create base and event
$base = event_base_new();
$event = event_new();

// Allocate a new context
$context = new ZMQContext();

// Create sockets
$rep = $context->getSocket(ZMQ::SOCKET_REP);

// Connect the socket

// Get the stream descriptor
$fd = $rep->getsockopt(ZMQ::SOCKOPT_FD);

// set event flags
event_set($event, $fd, EV_READ | EV_PERSIST, "print_line", array($rep, $base));

// set event base
event_base_set($event, $base);

// enable event

// start event loop



// Assign socket 1 to the queue, send and receive
var_dump($queue->send("hello there!")->recv());

You will notice when you run it that the server gets a couple of events that are not actually incoming messages. Right now ZeroMQ doesn't expose the nature of these events, but they are the socket initialization and client connect. You will also get one for the client disconnect. A future version of the ZeroMQ library will expose these so you can properly catch when clients connect to your server.

There really isn't much else to say. The code should be self-explanatory. If not, see the PHP libevent docs and the PHP ZeroMQ docs. And if you build something cool with this, please let me know.

ASRock Sandy Bridge Motherboard notes
I have pieced together two Sandy Bridge machines. This entry contains my notes on the two machines. Mostly for myself to refer back to later, but it might come in handy for others along the way.

Machine 1 - Overkill HTPC

  • Mythbuntu 10.10 initially but upgraded to full 11.04 when it was released
  • i5-2500k CPU
  • ASRock H67M LGA 1155 Intel H67 HDMI SATA 6Gb/s USB 3.0 Micro ATX Intel Motherboard
  • Seasonic PSU
  • G.SKILL Ripjaws X Series 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Model F3-10666CL9D-8GBXL
  • Crucial RealSSD C300 CTFDDAC064MAG-1G1 2.5" 64GB SATA III MLC SSD
  • Western Digital Caviar Green WD20EARS 2TB SATA 3.0Gb/s 3.5" HD
  • ASUS ENGT430/DI/1GD3(LP) GeForce GT 430 (Fermi) 1GB 128-bit DDR3 PCI Express 2.0 x16 HDCP Graphics card
  • AVS Gear GP-IR01BK Windows Vista Infrared MCE Black Remote Control
  • SilverStone Aluminum/Steel Micro ATX HTPC Computer Case GD05B (Black)
  • SiliconDust HDHomeRun HDHR-US Dual Tuner
  • RCA ANT751 Outdoor Antenna (installed in attic - see

Machine 2 - Dev Box for the office

  • Ubuntu 11.04
  • i7-2600k CPU
  • ASRock Z68 Extreme4 LGA 1155 Intel Z68 HDMI SATA 6Gb/s USB 3.0 ATX Intel Motherboard
  • G.SKILL Ripjaws X Series 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Model F3-10666CL9D-8GBXL
  • G.SKILL Ripjaws Series 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) Model F3-12800CL9D-8GBRL
  • Crucial M4 CT128M4SSD2 2.5" 128GB SATA III MLC Internal Solid State Drive (SSD)
  • 2 x SAMSUNG Spinpoint F4 HD204UI 2TB 5400 RPM SATA 3.0Gb/s 3.5" HD
  • CORSAIR Builder Series CX430 CMPSU-430CX 430W ATX12V Active PFC PSU
  • Old Antec case I had lying around

I went scouring slickdeals and other deal sites for most of these components, so there are some mismatches. Like the slightly mismatched ram in the second machine, and the fact that I am using a 2500k in an H67 (B2!) board. No real point in an unlocked cpu in a locked board, but the k was cheaper than the non-k at the time, and who knows, I could swap the motherboard. And yes, it is a B2-stepping board, so the SATA2 ports are iffy. But since I am not using them it doesn't bother me.

Continue reading "ASRock Sandy Bridge Motherboard notes"

Writing an OAuth Provider Service
Last year I showed how to use pecl/oauth to write a Twitter OAuth Consumer. But what about writing the other end of that? What if you need to provide OAuth access to an API for your site? How do you do it?

Luckily John Jawed and Tjerk have put quite a bit of work into pecl/oauth lately and we now have full provider support in the extension. It's not documented yet at, but there are some examples in svn. My particular project was to hook an OAuth provider service into a large existing Kohana-based codebase. After a couple of iterations this should now be trivial for others to do with the current pecl/oauth extension.

Continue reading "Writing an OAuth Provider Service"

A quick look at XHP
Facebook released a new PHP extension today that supports inlining XML. This is a feature known as XML Literals in Visual Basic. Go read their description here: It adds an extra parsing step which maps inlined XML elements to PHP classes. These classes are core.php and html.php which covers all the main HTML elements. The syntax of those class definitions is a bit odd. That oddness is explained in the How It Works document. Essentially, it lets you turn: Hello, {$_POST['name']}."; } else { ?>
What is your name?
<?php } into: Hello, {$_POST['name']}.; } else { echo
What is your name?
; } The main interest, at least to me, is that because PHP now understands the XML it is outputting, filtering can be done in a context-sensitive manner. The input filtering built into PHP can not know which context a string is going to be used in. If you use a string inside an on-handler or a style attribute, for example, you need radically different filtering from it being used as regular XML PCDATA in the html body. Some will say this form is more readable as well, but that isn't something that concerns me very much. The real question here is what is this runtime xml validation going to cost you. I have given talks in the past where I have used "class br extends html { ... }" as a classic example of something you should never do. A br tag is just a br tag. When you need one, stick a
in your page, don't instantiate a class and call a render() method. So, when I looked at html.php and saw: class :br extends :xhp:html-singleton { category %flow, %phrase; protected $tagName = 'br'; } I got a bit skeptical. Another thing I have been known to tell people is, "Friend don't let friends use Singletons." Which isn't something I came up with. Someone, a friend, I guess, told me that years ago. Ok ok, as Marcel points out in the comments, this isn't a real singleton, just in name. The "singleton" looks like this: abstract class :xhp:html-singleton extends :xhp:html-element { children empty; protected function stringify() { return $this->renderBaseAttrs() . ' />'; } } which extends html-element which in turn extends primitive. You can go read all the code for those yourself. Note that to build XHP you will need flex 2.5.35 which most distros won't have installed by default. Grab the flex tarball and ./configure && make install it. Then you are ready to go. I pointed Siege at my rather underpowered AS1410 SU2300 with the above trivial form examples. The plain PHP one and the XHP version. Ran each one 5 times benchmarking for 30s each time. The plain PHP one averaged around 1300 requests/sec. Here is a representative sample: acer:~> siege -c 3 -b -t30s http://xhp.localhost/1.php ** SIEGE 2.68 ** Preparing 3 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 38239 hits Availability: 100.00 % Elapsed time: 29.60 secs Data transferred: 3.97 MB Response time: 0.00 secs Transaction rate: 1291.86 trans/sec Throughput: 0.13 MB/sec Concurrency: 2.93 Successful transactions: 38239 Failed transactions: 0 Longest transaction: 0.05 Shortest transaction: 0.00 And the XHP version: Transactions: [...]

HipHop PHP - Nifty Trick?
In a response to a question from ReadWriteWeb, among other things, I wrote:
My main worry here is that people think this is some kind of magic bullet that will solve their site performance problems. Generating C++ code from PHP code is a nifty trick and people seem to have gotten quite excited about it. I'd love to see those same people get excited about basic profiling and identifying the most costly areas of an application. Speeding up one of the faster parts of your system isn't going to give you anywhere near as much of a benefit as speeding up, or eliminating, one of the slower parts of your overall system.
The "nifty trick" part of that seems to have become the story, and them injecting a "just" in front it of it makes it sound more derogatory. Anyone who knows me knows that I am a big fan of nifty tricks that solve the problem. When I first heard about the Facebook effort I was assuming they were writing a JIT based on LLVM V8 or something along those lines. Writing a good JIT is hard. Doing static code analysis and generating compilable C++ from it is indeed a nifty trick. It's not "just" a nifty trick, it is a cool trick that takes advantage of a number of characteristics of PHP. The main one being that you can't overload PHP functions. strlen() is always strlen, for example. In Python, this would be harder because you can overload everything.

I also noted that most sites on the Web have a lot of lower hanging fruit that would provide a much bigger performance improvement, if fixed, than doubling the speed of the PHP execution phase. The ReadWriteWeb site, for example, needs 160 separate HTTP requests and 41 distinct DNS lookups to load the front page. And once you get beyond the frontend inefficiencies you usually find Database issues, inefficient system call issues and general architecture problems that again aren't solved by speeding up PHP execution.

If you have done your homework and find that your web servers are cpu-bound, you are already using an opcode cache like APC and your Callgrind callgraph shows you that the PHP executor is a significant bottleneck, then HipHop PHP is definitely something you should be looking at.

SQLi Detection - Duh Moment
Not sure why it took me so long to figure out what I am sure is obvious to most other people who have thought about this, but it never clicked for me how to get anywhere near useful SQL Injection detection. The injection itself is trivial, of course, but determining whether it actually worked and weeding out false positives in an automated manner was something that seemed too hard.

During my run on Friday I had a Duh! moment on it. Annoyingly simple. Do it in 3 requests. Request #1 is a normal request. For example, "?id=1" in the URL. If the id is being passed to an SQL request it will return a single record or perhaps no record, it doesn't really matter. Now on request #2 do "?id=1 or 3=4", that is, inject a false 'OR' condition. If the output changes, we are done. Nothing to see here. However, if the output does not change we send request #3 with "?id=1 or 3=3" and if that output differs from request #2 then we have a potential SQLi situation. There are of course still chances of false positives (and negatives) with page stamps and such, but filtering out the response headers and html comments cuts down on that a bit. Add different combinations of single and double-quotes, like "?id=1'or'3'='3" (without the double-quotes, of course) and it might be able to catch something.

The best thing about it is that it can slide into an existing scanner framework quite easily. If you have a base reference request, then it just adds a single request to the common case where the false 'OR' condition output does not match the base reference. You only need to do the true 'OR' condition request in case it does match.

Anybody have any other approaches?

Playing with Gearman
This was written in September 2009 when the current version of Gearman was 0.9. Thanks to Eric Day for answering my dumb questions along the way.

To get started, install Gearman. I am on Debian, so this is what I installed:
% apt-get install gearman gearman-job-server gearman-tools libgearman1 libgearman-dev libdrizzle-dev
Enable Gearman in /etc/default/gearman-server
Set up Gearman to use MySQL for its persistent queue store in /etc/default/gearman-job-server
 PARAMS="-q libdrizzle --libdrizzle-host= --libdrizzle-user=gearman \
                       --libdrizzle-password=your_pw --libdrizzle-db=gearman \
                       --libdrizzle-table=gearman_queue --libdrizzle-mysql"

% mysqladmin create gearman

% mysql 
mysql> create USER gearman@localhost identified by 'your_pw';
mysql> GRANT ALL on gearman.* to gearman@localhost;
** Careful, if you are running MySQL using --old-passwords this won't work with libdrizzle. You will need to get the 41-char password hash with a little snippet of PHP that does the double sha1 encoding:
% php -r "echo '*'.strtoupper(sha1(sha1('your_pw',true)));"

% mysql
mysql> UPDATE mysql.user set Password='above_output' where User='gearman';

% mysqladmin flush-privileges
Continue reading "Playing with Gearman"