samadhiweb

smalltalk programming for the web



Creating an X.509 certificate request

6 December 2017

From within Pharo:

| rsa |
rsa := LcRSA generateKey: 2048.
LcX509Request new
  setSubject: 'www.samadhiweb.com';
  setPublicKey: (LcEvpPublicKey setRSA: rsa);
  sign;
  asString

The output is an X.509 certificate request, suitable for Let's Encrypt:

Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: CN=www.samadhiweb.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b6:c4:94:dc:83:4c:8f:92:6c:8c:f5:e4:3a:66:
                    b9:6a:90:e8:51:aa:58:da:e4:ea:3a:59:1d:78:96:
                    14:ab:83:23:2f:20:45:13:70:88:8d:b5:93:8c:18:
                    c5:78:70:c9:b3:8e:e3:f4:66:64:9c:9d:78:b9:dc:
                    9b:2b:e1:6d:06:d0:b4:6a:83:94:4e:6e:27:dd:12:
                    7f:79:b2:a1:67:ed:96:bb:29:fd:dd:dd:78:5a:8c:
                    c1:8f:83:8f:f7:c8:b1:52:d2:8b:de:9f:6b:31:9c:
                    1f:b7:99:34:83:c1:e6:9d:d4:10:59:ff:bd:32:9f:
                    0a:8c:e2:3b:06:8c:6c:60:24:e2:89:6c:1f:d4:97:
                    c0:e2:77:44:98:31:d8:3f:41:29:e0:ac:ef:56:fb:
                    7c:a5:55:94:45:51:3d:b6:4e:5b:e7:1b:05:7c:7e:
                    0b:1d:cc:a3:88:30:7a:98:51:c7:64:cf:d6:29:f1:
                    16:86:00:b4:72:10:1a:c7:77:16:9d:94:c3:4e:fc:
                    83:63:e5:d7:09:d6:40:cb:d4:26:7e:3e:c7:e4:64:
                    7f:7e:1c:af:53:68:24:f5:5c:a6:79:76:2f:cf:29:
                    bf:9c:3d:3e:38:00:5e:9d:a6:f3:92:09:9b:eb:e4:
                    b4:24:ca:a3:d9:ed:6e:ed:bd:06:af:40:fe:7b:f3:
                    50:c7
                Exponent: 65537 (0x10001)
        Attributes:
            a0:00
    Signature Algorithm: sha256WithRSAEncryption
         4a:21:79:18:51:92:2c:d7:cd:ac:f0:f3:bc:6e:53:f3:de:72:
         11:56:ce:f3:e1:71:61:a5:8a:cb:95:4f:31:36:1e:65:50:09:
         39:71:95:a2:a5:30:c6:fc:36:29:7f:6e:86:1f:2a:52:a9:b8:
         55:39:b2:82:b6:67:84:b3:35:2b:be:57:5f:fd:7e:a5:b1:b9:
         ab:20:65:d7:20:68:97:14:0f:13:30:1d:54:c9:7f:55:0d:58:
         0a:14:2d:5b:80:57:a0:6a:73:31:6d:64:52:43:d1:6d:c2:3b:
         8f:88:25:9a:6e:50:52:1d:f8:66:c0:d7:ed:85:be:ef:8b:63:
         20:c1:48:e4:d2:30:6c:cf:07:27:bb:a7:56:82:8a:7d:c7:fe:
         73:29:f1:4a:f4:18:77:0e:56:48:20:41:08:e5:2e:2c:07:67:
         16:e3:9c:d5:c3:0e:02:17:c4:11:5c:01:19:20:4b:c9:e2:37:
         3b:14:02:90:77:d2:17:1d:62:24:cc:5b:1c:b0:c9:48:71:02:
         c6:ff:d8:5e:57:30:bb:da:c7:73:4d:40:ed:b2:8f:a1:2c:07:
         d6:58:22:94:7f:ff:bc:94:38:23:30:fb:0c:51:53:6c:13:f6:
         dc:92:9e:c8:67:fa:ed:3b:e7:1a:1d:00:6e:5c:f6:c5:27:7c:
         d7:76:5a:78

OpenSSL wrapper for Pharo

29 October 2017

I've put up the beginnings of a wrapper for OpenSSL on STH:

Metacello new
  baseline: 'OpenSSL';
  smalltalkhubUser: 'PierceNg' project: 'OpenSSL-Pharo';
  load.

Verified on Pharo 6 32- and 64-bit.

My near term goal is to wrap enough libcrypto functionality to implement the client-side of Let's Encrypt.

I meant to put it up on GH, for the ease of forking and PRs, but I couldn't get Iceberg to work, and gitfiletree also failed to load, so STH it is for now.

Collaboration welcome.

Ubuntu LVM root filesystem full

20 August 2017

I had carelessly allowed the root filesystem on one of my Ubuntu servers to become full. Here's how I recovered from it.

Setup: Ubuntu server 16.04, entire disk for LVM, all disk space assigned to one volume group (VG), root (4GB) and swap (1.5GB) in their own logical volumes (LV), with the remaining disk space allocated to LXD containers using LVM thin provisioning.

The root filesystem, from here on referred to as root if the context is clear, had become full because I hadn't been purging old Linux kernels. Being full, the filesystem was then unable to complete "apt-get autoremove". I needed to add some space to root, which I took from swap.

Following commands were performed online with the server in multi-user mode. First, turn off swap.

# swapoff -a

Then, reduce the size of swap LV by 1GB.

# lvresize -L-1G /dev/srv-vg/swap_1

Add the 1GB taken to root; the command grows the root LV and resizes the filesystem within the LV.

# lvextend -L+1G /dev/srv-vg/root -r -v

"df -k" shows that root now has an additional 1GB and is no longer full. Fix any broken apt dependencies and purge the old Linux kernels.

# apt-get -f install
# apt-get autoremove

"df -k" now shows that root has even more free space.

Now I set about to give the 1GB back to swap. While it is possible to grow root's LV and filesystem online, shrinking root needs to be done offline by booting from live DVD into single user mode at the server console. (The server runs virtually somewhere "in the cloud". Console access is via VNC. I use the Linux client Vinagre. Obligatory Smalltalk content: Vinagre is also the client I use to connect to RFBServer running in the Pharo image that runs this blog.)

For below commands, I'm at single user mode, booted from live DVD, with all server filesystems unmounted. First, check the root filesystem.

# e2fsck -f /dev/srv-vg/root

Remember the root filesystem is contained within the root LV. Now resize the filesystem. Here "4G" refers to the absolute size.

# resize2fs /dev/srv-vg/root 4G

Next, resize the root LV.

# lvresize -L-1G /dev/srv-vg/root

Check root filesystem again.

# e2fsck -fy /dev/srv-vg/root

Now, give 1GB back to swap.

# lvresize -L+1G /dev/srv-vg/swap_1

Reboot into multi-user mode, turn swap back on, and look around.

This incident happened to the server that is running this blog. That you are now reading this blog post means that the above procedure succeeded. :-P

DataFrame and Simple Linear Regression

6 August 2017

In the previous post I mentioned using Iceberg successfully. The code I was pushing is SLRCalculator, a simple linear regression calculator, written to take Oleksandr Zaytsev's DataFrame library for a spin, by way of porting Jason Brownlee's excellent simple linear regression in Python to Pharo.

Firstly, install DataFrame. This also pulls in Roassal.

Metacello new
  baseline: 'DataFrame';
  repository: 'github://PolyMathOrg/DataFrame';
  load.

SLRCalculator implements mean, variance, covariance, coefficients etc, and also incorporates the Swedish automobile insurance dataset used by Jason in his Python example.

SLRCalculator class>>loadSwedishAutoInsuranceData
  "Source: https://www.math.muni.cz/~kolacek/docs/frvs/M7222/data/AutoInsurSweden.txt"
  | df |

  df := DataFrame fromRows: #(
            ( 108 392.5 )
            ( 19 46.2 )
            ( 13 15.7 )
            "more lines" ).
  df columnNames: #(X Y).
  ^ df

The computation for covariance also uses DataFrame.

covariance: dataFrame
  | xvalues yvalues xmean ymean covar |

  xvalues := dataFrame columnAt: 1.
  yvalues := dataFrame columnAt: 2.	
  xmean := self mean: xvalues.
  ymean := self mean: yvalues.	
  covar := 0.
  1 to: xvalues size do: [ :idx |
    covar := covar + (((xvalues at: idx) - xmean) * ((yvalues at: idx) - ymean)) ].
  ^ covar 

Let's see how to use SLRCalculator to perform linear regression, with graphing using Roassal. First declare the variables and instantiate some objects:

| allData splitArray trainingData testData s coeff g dsa dlr legend |

s := SLRCalculator new.

allData := SLRCalculator loadSwedishAutoInsuranceData.

Next, split the data set into training and test subsets. Splitting without shuffling means to always take the first 60% of the data for training.

splitArray := s extractForTesting: allData by: 60 percent shuffled: false.  
trainingData := splitArray at: 1.
testData := splitArray at: 2.
coeff := s coefficients: trainingData.

Set up for graphing. Load `allData' as points.

g := RTGrapher new.

allData do: [ :row |
  dsa := RTData new.
  dsa dotShape color: Color blue.
  dsa points: { (row at: 1) @ (row at: 2) }.
  dsa x: #x.
  dsa y: #y.
  g add: dsa ].

Create the points to plot the linear regression of the full data set, using the coefficients computed from the training subset.

dlr := RTData new.
dlr noDot.
dlr connectColor: Color red.
dlr points: (allData column: #X).
" y = b0 + (b1 * x) "
dlr x: #yourself.
dlr y: [ :v | (coeff at: 1) + (v * (coeff at: 2)) ].
g add: dlr.

Make the plot look nice.

g axisX noDecimal; title: 'Claims'.
g axisY title: 'Total payment in SEK'.
g shouldUseNiceLabels: true.
g build.

legend := RTLegendBuilder new.
legend view: g view.
legend addText: 'Swedish Automobile Insurance Data Linear Regression'.
legend build.

g view

Putting the code altogether:

| allData splitArray trainingData testData s coeff g dsa dlr legend |

s := SLRCalculator new.

allData := SLRCalculator loadSwedishAutoInsuranceData.
splitArray := s extractForTesting: allData by: 60 percent shuffled: false.  
trainingData := splitArray at: 1.
testData := splitArray at: 2.
coeff := s coefficients: trainingData.

g := RTGrapher new.

allData do: [ :row |
  dsa := RTData new.
  dsa dotShape color: Color blue.
  dsa points: { (row at: 1) @ (row at: 2) }.
  dsa x: #x.
  dsa y: #y.
  g add: dsa ].

dlr := RTData new.
dlr noDot.
dlr connectColor: Color red.
dlr points: (allData column: #X).
" y = b0 + (b1 * x) "
dlr x: #yourself.
dlr y: [ :v | (coeff at: 1) + (v * (coeff at: 2)) ].
g add: dlr.

g axisX noDecimal; title: 'Claims'.
g axisY title: 'Total payment in SEK'.
g shouldUseNiceLabels: true.
g build.

legend := RTLegendBuilder new.
legend view: g view.
legend addText: 'Swedish Automobile Insurance Data Linear Regression'.
legend build.

g view

Copy/paste the code into a playground, press shift-ctrl-g...