The Big Browser

M y    b r a i n    h u r t s  !                                           w e                 r e a l l y                 t h i n k                   w h a t                y o u             k n o w

29 November 2020

Convert all MOV video files in the current folder to MP4

Convert a single MOV video file to MP4 format:

  ffmpeg -i my_video.mov my_video.mp4
Convert all MOV videos in the current folder and sub-folders to MP4 and delete the orignal .MOV files: For Linux or Mac, or on Windows under msys, cygwin, cmder etc.
for video in `find . -iname "*.mov"` ; do
    ffmpeg -i "${video}" "${video%.*}.mp4" && rm "${video}"
done
If ffmpeg is not present on your system install it with apt install ffmpeg (on Linux) or brew install ffmpeg on Mac.

22 November 2020

CSS Style Sheet HTML Preview Generator (python script)

How do I see an HTML preview of every single CSS rule defined in a CSS file?

There is no fixed way to do it, especially if your CSS style sheets make abundant use of absolute and fixed positioning. This script below is a quick and dirty attempt to generate HTML tags for all the rules defined in a CSS file. If the rule is defined for a known HTML tag then that tag is rendered as itself, otherwise a div tag is created.

The script does not cover every possible scenario and only supports the most common CSS constructs. It wraps all fixed CSS rules into a dedicated iframe in order to prevent elements from overlapping. You can disable wrapping fixed elements into frames by passing --no-frames.

If you have a large CSS file the HTML generation can take up to several minutes since nested rules are iterated recursively.

⚠️ For complex CSS style sheets (1MB+) the generated HTML can cause the browser to freeze or rendering to take a signficant time even on a performant computer. You have been warned!

Dependencies

The CSS preview generator depends on the cssutil python module which can be installed with:

pip3 install cssutils

You are welcome to fork, improve and provide feedback.

Usage

css_stylesheet_preview_generator.py mystyles.css -o mypreview.html
For more options run with the -h or --help switch.

Example output

Python CSS preview generator script

16 October 2017

Barcelona city bikes availability: a console REST client with curl and jq

How do you consume a REST api in the terminal? All we can do is just curl a REST end point which will spit some uglified JSON into the console, right?

Let's say we need to get a list of public bike stations in Barcelona:

curl -s https://bicing.barcelona/get-stations


Hmmm, that's not very readable... Can we do better than this?

Yes, we can!

Meet jq — a full-fledged console JSON parser, as powerful as awk but for JSON.

Available in standard linux repos:

sudo apt install jq

as well as in homebrew on Mac:

brew install jq

Now we can display the list of all bike stations in Barcelona with JSON pretty-printing and colorization:

curl -s https://bicing.barcelona/get-stations | jq

Now, this is more readable. But how about filtering for specific bike stations, say the ones closest to our daily destinations?

We look up the station IDs we need at the offical Barcelona bicing
site
and pass the station IDs as parameters to jq:

curl -s https://bicing.barcelona/get-stations --compressed | jq -r '.stations[] | select(.id==("153", "154", "339", "382","150"))'

Since we don't want all the fields in the JSON, we pass in the JSON keys for the data we want to keep.

Get bikes by station IDs, and extract station address, number of bikes, and number of parking slots:

curl -s https://bicing.barcelona/get-stations --compressed | jq -r '.stations[] | select(.id==("153", "154", "339", "382","150")) | .streetName,.bikes,.slots'

This looks a bit better, but by default jq prints one key per line, and we want to add some formatting, so that data for one station shows in a single line. We can use jq string interpolation to achieve this. Here is the final version which prints our data in a tabular format (available bikes, parking slots, station id, and address):

echo 'AVL\tEMP\tSID\tADR'
curl -s https://www.bicing.barcelona/get-stations --compressed | jq -r '.stations[] | select(.id==("153", "154", "339", "382","150")) | "\(.bikes)\t\(.slots)\t\(.id)\t\(.streetName), \(.streetNumber)"'

And here is the final result:

And, finally: what's the point of adding the --compressed parameter to curl?

Setting --compressed is equivalent to setting --header 'Accept-Encoding: gzip,deflate' — as a result the server will compress the output with gzip or deflate (if capable of compression), and curl will decompress the response on the fly, which will make the result appear faster since less bytes will be sent over the network.

13 July 2016

Simple webserver with twisted

The Python oneliner webserver python -m SimpleHTTPServer (for Python 2.x) or python -m http.server (for Python 3.x) is useful in many scenarios, but sometimes it simply does not cut it because it is meant for development only, is not secure, and is single-threaded.

A better oneline webserver comes with Twisted, the library that the original bittorrent reference implementation was written in.
Installing twisted for the current user only:

pip install twisted --user

Installing twisted system-wide:

sudo pip install twisted

Once installed you can run the default twisted webserver with a command like twistd -n web --path . where the dot represents the current folder. Replace with an absolute path if you need.

The version below additionally displays the internal and external IP addresses, so that you can launch the server and instantly connect to it by entering the displayed IP on your mobile devices or another computers in the local network.

echo 'IP Address:' $(ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'):8080
echo 'External IP:' $(wget http://ipinfo.io/ip -qO -):8080
twistd -n web --path .

In order to be able to connect via the external IP you'll need to set up port forwarding on your router as described at portforward.com.

03 March 2016

Creating native dictionaries for the OSX Dictionary.app

OSX has an integrated dictionary app which can be accessed by selecting some text in any application, right-clicking and then picking Look up from the context menu (three-finger-tap on a selected word should work too).

The OSX Dictionary comes pre-installed with several dictionaries but creating your own dictionaries or converting existing ones is surprisingly complex.

This post is an attempt to make this process easier for everyone.

Most third-party dictionaries that I use are in DSL (ABBY Lingvo) and BGL (Babylon) formats. I have collected them over the years, and have been using them both on the desktop via the open-source app GoldenDict, and on my smartphone thanks to GoldenDict for android. The OSX version of GoldenDict is still buggy, and apparently not properly maintained, so I wanted to find a way to migrate my existing dictionaries to the native OSX dictionary app.

After a bit of research I finally figured out the toolchain for converting my existing dictionaries in Babylon and Lingvo DSL formats to the AppleDict format which the OSX Dictionary.app expects.

Prerequisites

Auxiliary Tools for Xcode

Download the DMG for Auxiliary Tools for Xcode from developer.apple.com/download/more. You'll need to sign in with your Apple ID in order to download. If you already have an iCloud account then you can use it. Once you sign in you can use the direct link to Auxiliary Tools for Xcode 7 or use the latest version by searching for it on the Apple downloads page.

Mount the DMG file by double-clicking it in Finder. Next, As the root user create a folder at /Developer/Extras and copy the Dictionary Development Kit folder from the Auxiliary Tools into that folder.
sudo mkdir -p /Developer/Extras
sudo cp -r "/Volumes/Auxiliary Tools/Dictionary Development Kit" /Developer/Extras

Python 3

Install Python 3 via homebrew (OSX comes with Python 2.x preinstalled):
brew cask install python3

Python 3 dependencies

Install lxml and BeautifulSoup, the parsers that pyglossary depends on:
## for current user only
pip3 install lxml beautifulsoup4 --user

## system-wide, for all users
sudo pip3 install lxml beautifulsoup4

PyGlossary

Clone the pyGlossary project to a folder on your drive:
mkdir -p ~/projects
cd ~/projects
git clone --depth 1 git@github.com:ilius/pyglossary.git
Next, locate the path to pyglossary.pyw — that's the script you'll need for converting the DSL file in UTF-8 encoding to AppleDict XML.

DSL to AppleDict conversion

The DSL to AppleDict sequence:
  1. DSL files come in Unicode Little Endian encoding, so the first step would be converting your DSL file from UTF-16 to UTF-8:
  2. iconv -f UTF-16 -t UTF-8 webster-original-utf16.dsl > webster.dsl
    
  3. Now do the actual conversion to AppleDict source format. Make sure you include the correct path to pyglossary.pyw. When the command completes (which might take several minutes if the source file is big), a new file webster.xml will be created along with some make files.
  4. ~/projects/pyglossary/pyglossary.pyw --read-format=ABBYYLingvoDSL --write-format=AppleDict webster.dsl webster.xml
    
  5. The generated AppleDict source file will be created in a subfolder with the same name as the source file, i.e. webster. Next, we move into that folder and do the compilation of AppleDict source to the OSX binary dictionary format:
  6. cd webster
    make
    make install
    
Running make install will copy the compiled dictionary to ~/Library/Dictionaries. Restart the Dictionary.app, open preferences (⌘ + ,), and you should see the new dictionary available in the list. Enable the corresponding checkbox to make the dictionary active.

Here is a bash script that automates the process of DSL to AppleDict conversion: https://gist.github.com/glowinthedark/da0a026ce0fb691e418c306abc67e525. Run the bash script without arguments to see usage notes.


IMPORTANT: You will need to edit the script and set the correct value of PYGLOSSARY_HOME, by default it is ~/projects/pyglossary.

Babylon BGL to AppleDict conversion

The conversion steps for BGL dictionaries:



A bash script for BGL to AppleDict is available at https://gist.github.com/glowinthedark/cbb9941f72ae1776b9fbbfff39666ce7. To get this toolchain to work I recommend reading the readme files which are included both with pyglossary and the Apple Dictionary Development Kit. I have included the most important points in the help section of the bash script for DSL.

Where to get the dictionaries?

There are literally hundreds, maybe thousands of dictionaries in DSL, Stardict, and Babylon BGL formats. A search engine query with the format and language pairs will most likely return some links where you'll be able to download the files. For example try stardict+english+spanish. The Goldendict dictionaries download page also has some useful links.

There are also a couple of command line utilities for converting between free dictionary formats: dictconv, makedict.