hide's memo
25 Mar, 2023

Align images from right to left

[Japanese(日本語)]

How to align images from right to left.

 

 

 

 

 

 

 

 

 

 

 

 

 

you can set “rtl” to “p” tag’s “dir” attribute.

<html>
<body>
<p dir="rtl">
<img src="001.png">
<img src="002.png">
<img src="003.png">
<img src="004.png">
<img src="005.png">
<img src="006.png">
<img src="007.png">
</p>
</body>
</html>
18 Mar, 2023

adidas ADIZERO BOSTON 11

[Japanese(日本語)]

I bought an adidas ADIZERO BOSTON 11 on march 2023.
But, when I run with these shoes, my instep of my right foot hurts.

Even loosening the shoelace did not help.
the shoe tongue of this shoe is connected with the sole and it is very tight.
finally I cut the tongue of this shoe, and now, the shoes are very confortable for me.

16 Mar, 2023

convert csv to json

Sample src of How to convert csv to json.
You can change csv file to json.
You can also add keys, convert string to integer, convert string to boolean.

[Japanese(日本語)]

 

import csv
import json
import sys

#---------------------------------------
# change string(0/1) to Boolean(False/True)
#---------------------------------------
def to_boolean(s):
    if(s == ""):
        return None
    if(s == "0"):
        return False
    elif(s=="1"):
        return True
    else:
        raise ValueError("not boolean")


#---------------------------------------
# main
#--------------------------------------
args = sys.argv
jdata = {"field01":"hello", "filed02":"world", "country_records":[]}
r = open(args[1],"r")

d_reader = csv.DictReader(r)
d_list = [row for row in d_reader]

for row in d_list:
    row["population"] = int(row["population"])
    row["capital"] = to_boolean(row["capital"])

jdata["country_records"] = d_list
print(json.dumps(jdata, ensure_ascii=False, indent=2))
25 Feb, 2023

Print out excel file by VBScript

[Japanese(日本語)]

This is a sample of how to print out excel file’s cells by VBScript.
https://github.com/hidemiubiz/public/blob/main/VBS/excelparse.vbs

'-------------------------------------------------------------------
' Excelファイルの中で、文字列が入ったセルから文字列を抜き出すサンプル
' this is a sample extracting string from excel files.
'
' Usage: csript excelparse.vbs input.xls left-top-cell right-bottom-cell
' Example:
' If you want to print sample.xls's B5 to E7 celss, put the followig command.
' cscript excelparse.vbs sample.xls B5 E7
'-------------------------------------------------------------------

set objXls = CreateObject("Excel.Application")
objXls.Visible=False
objXls.ScreenUpdating=False

Dim filename
Dim left_top_cell
Dim right_bottom_cel
filename = Wscript.Arguments(0)
left_top_cell = Wscript.Arguments(1)
right_bottom_cell = Wscript.Arguments(2)

call ParseExcel(filename, left_top_cell, right_bottom_cell)

'-----------------------------------------------------
' 標準出力(Print stdout)
'-----------------------------------------------------
Function Print(m)
WScript.Stdout.Write m
End Function

'-----------------------------------------------------
' Get Full path
'-----------------------------------------------------
Function GetFullPath(ByRef filename)
GetFullPath = CreateObject("Scripting.FileSystemObject").GetAbsolutePathName(filename)
End Function

'-----------------------------------------------------
' Excelファイルをパースして出力
' print out excel file
' filename .............. excel filename パースするExcelファイル
' left_top_cell ......... left_top_cell string(example ... "B5")
' right_bottom_cell ..... right_bottom_cell string (such as "D10")
'-----------------------------------------------------
Sub ParseExcel(ByRef filename, ByRef left_top_cell, ByRef right_bottom_cell)
Dim wb,s
Dim lef_top_row
Dim lef_top_col
Dim right_bottom_row
Dim right_bottom_col

Set wb = objXls.Workbooks.Open(GetFullPath(filename))
Dim i
For i=1 To wb.Sheets.Count
Set s = wb.Sheets(i)
Set lt = s.Range("A1:" & left_top_cell)
Set rb = s.Range("A1:" & right_bottom_cell)
left_top_row = lt.Rows.Count
left_top_col = lt.Columns.Count
right_bottom_row = rb.Rows.Count
right_bottom_col = rb.Columns.Count
Call PrintRange(s, left_top_row, right_bottom_row, left_top_col, right_bottom_col)
Next
wb.Close
End Sub

'-----------------------------------------------------
' 指定された範囲のセルの文字列を表示する。
' print out specific range cells
' sheet .......... シートObject
' startRow ....... 行の開始位置(left_top_row)
' endRow ......... 行の終了位置(right_bottom_row)
' startCol ....... 列の開始位置(left_top_col)
' endCol ......... 列の終了位置(right_bottom_col)
'-----------------------------------------------------
Sub PrintRange(ByRef sheet, ByVal startRow, ByVal endRow, ByVal startCol, ByVal endCol)
Dim y
Dim x
For y=startRow To endRow
For x=startCol To endCol
Set c = sheet.Cells(y,x)
Print(c.Text & ",")
Next
Print(vbCrLf)
Next
End Sub

 

■How to use

if you want to print out from B5 cell to E7 cell like below image.
cscript  thissample.cbs  input.xlsx  B5  E7

 

31 Jul, 2022

How to UPSERT using Spark Dataframe

[Japanese(日本語)]

I am not sure if this article has any problem. If you found something wrong in this article. please let me know.

you have 2 csv data files and want to upsert as follows.

■Sampe Code No.1

df1 = spark.read.option("header","true").csv("data01.csv")
df2 = spark.read.option("header","true").csv("data02.csv")

df = df1.join(df2, ['Country','City'], 'left_anti')
df = df.unionAll(df2)

df.write.option("header","true").format("csv").save("output")

 

■Sample Code No.2
Another example(this code is sometimes work intensionally, But sometimes doesn’t work intensionally.
It depends on how many partitions the program execute)

df1 = spark.read.option("header","true").csv("data01.csv")
df2 = spark.read.option("header","true").csv("data02.csv")

df = df2.unionAll(df1)
df = df.dropDuplicates(['Country','City'])

df.write.option("header","true").format("csv").save("output")

 

 

■Sample Code No.3
this code will probably work intensionally(sorry, no guarantee).

df1 = spark.read.option("header","true").csv("data01.csv")
df2 = spark.read.option("header","true").csv("data02.csv")

df = df2.unionAll(df1)
df = df.dropDuplicates(['Country','City'])

df=df.coalesce(1) 
df.write.option("header","true").format("csv").save("output")

 

■Difference between No2. and No.3 code is the following place.

I think Spark is “Lazy Evaluation”. So, When you call the folowing code.

df = df2.unionAll(df1)
df = df.dropDuplicates(['Country','City'])

Spark Dataframe doesn’t work actually yet.It works when the following code was executed.

df.write.option("header","true").format("csv").save("output")

So, I guess the Sample Code No.3  forced Dataframe work with 1 partition,and then, dropDuplicaets() work intentionally.

28 May, 2022

Upload files to S3 Bucket using curl.

[日本語]

I followed the information below.
tuxfight3r/s3_upload.sh

#!/bin/bash

S3KEY="Your Acess Key"
S3SECRET="Your Access Secret"
S3BUCKET="Your Bucket Name"
S3STORAGETYPE="STANDARD"
AWSREGION="Your S3 Region"

# If you use mac OS. enable following code. 
#OS="mac"

function putS3
{
  file_path=$1
  aws_path=$2
  bucket="${S3BUCKET}"
  date=$(date -R -u)

  acl="x-amz-acl:private"
  content_type="text/plain"
  storage_type="x-amz-storage-class:${S3STORAGETYPE}"
  string="PUTnn$content_typen$daten$acln$storage_typen/$bucket$aws_path${file_path##/*/}"

  if [ $OS = "mac" ]; then
    signature=$(printf "${string}" | openssl sha1 -hmac "${S3SECRET}" -binary | base64)
  else
    signature=$(echo -en "${string}" | openssl sha1 -hmac "${S3SECRET}" -binary | base64)
  fi

  curl -s --retry 3 --retry-delay 10 -X PUT -T "$file_path" 
  -H "Host: $bucket.${AWSREGION}.amazonaws.com" 
  -H "Date: $date" 
  -H "Content-Type: $content_type" 
  -H "$storage_type" 
  -H "$acl" 
  -H "Authorization: AWS ${S3KEY}:$signature" 
  "https://$bucket.${AWSREGION}.amazonaws.com$aws_path${file_path##/*/}"

}

# Upload test.txt file to /test/ folder.
putS3 test.txt /test/
27 May, 2022

Extend HttpWebRequest Timeout

[日本語]

The default Timeout of HttpWebRequest is 100 seconds.
So, It will timeout if the server response take more than 100 seconds.

If you want to extend the Timeout. You can set The Timeout as your preferrd time.

But If you set the Timeout, You shoud call the timeout before calling “GetRequestStream()”

 

 

using System;
using System.Web;
using System.Net;
using System.Text;
using System.IO;

namespace ConsoleNetFramework
{
class Sample
{
static void Main(string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(“http://localhost/slow”);

request.Method=”POST”;
byte[] postData = System.Text.Encoding.ASCII.GetBytes(“message=hello”);

 request.Timeout = 3*60*1000; // <– You need to set Timeout here!!
                                                           //  before calling “GetRequestStream()”.

Stream writeStream = request.GetRequestStream();
 request.Timeout = 3*60*1000; // <– If you set Timeout here after calling “GetRequestStream()”.
// it doesn’t work.

writeStream.Write(postData, 0, postData.Length);

HttpWebResponse response;
// Get Response… and happen the TIMEOUT ERROR.
 response = (HttpWebResponse)request.GetResponse(); 

Stream receiveStream = response.GetResponseStream();
StreamReader sr = new System.IO.StreamReader(receiveStream, System.Text.Encoding.GetEncoding(“utf-8”));
Console.WriteLine(sr.ReadToEnd());
}
}
}

 

 

28 Apr, 2022

How to use PostgreSQL as a sink (Azure Data Factory)

[日本語]

This is sample of how to use PostgreSQL as a sink.

ADF can’t use PostgreSQL as a sink without using Self-hosted integration runtime.

So, at first, you need to set up a self-hosted runtime.
(See What is the Self Hosted Runtime(Azure Data Factory))

There is no PostgreSQL link service which can be used as a sink.
so you need to set “ODBC” link service.
You need to create the environment as below.

 

 

■Step1

Install PostgreSQL ODBC Driver as follows.

(1)Download PostgreSQL ODBC Driver
(1.1)access https://www.postgresql.org/
(1.2)”Download” -> “File Browser” -> “odbc” -> “versions” -> “msi”
(1.3)Download psqlodbc_13_02_0000-x64.zip (at first, I tried Version 9.2. but it didn’t work)

(2)Install ODBC Driver

 

■Step2

(1)Set sink’s dataset “ODBC”
(2)Set ODBC’s linked service as below.

17 Apr, 2022

DBD::mysql on MacBook Air(M1 macOS Monterey 12.0.1)

[日本語]

Actually, I did a lot of  try and error. so I am not sure the following instruction is sufficient.

You can also get information on the link below.
apache + perl + cgi + mysql (DBD::mysql, DBI) on MacBook M1 using Docker

 

(1)  brew install openssl
(2) add following setting on ~/.zshrc

export LIBRARY_PATH=/opt/homebrew/opt/openssl/lib
export C_INCLUDE_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/System/Library/Perl/Extras/5.30/darwin-thread-multi-2level/auto/DBI/
export CPLUS_INCLUDE_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/System/Library/Perl/Extras/5.30/darwin-thread-multi-2level/auto/DBI/

(3)source ~/.zshrc

(4)download and install DBI-1.643.tar.gz
(4.1)GET https://www.cpan.org/modules/by-module/DBI/DBI-1.643.tar.gz
(4.2)extract it on any directory you like.
(4.3)perl Makefile.PL
(4.4)make
(4.5)make install

(5) cpan install DBD::mysql

 

■Additional information

(1)Why you need to add some directory in ~/.zshrc

export LIBRARY_PATH=/opt/homebrew/opt/openssl/lib

-> If you don’t add the path above. you will encounter the following error, when “cpan install DBD::mysql”
“Can’t link/include C library ‘ssl’, ‘crypto’, aborting.”

export C_INCLUDE_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/System/Library/Perl/Extras/5.30/darwin-thread-multi-2level/auto/DBI/
export CPLUS_INCLUDE_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/System/Library/Perl/Extras/5.30/darwin-thread-multi-2level/auto/DBI/

-> If you don’t add the path above. you will encounter the following error.
‘DBIXS.h’ file not found

Why you need to install DBI-1.643.tar.gz
->I followed the information below.
https://stackoverflow.com/questions/68774807/dbdmysql-installation-on-catalina-big-sur-fail-with-extern-h-file-not-found

 

If you want to create the environment using Docker.

See the following link.

apache + perl + cgi + mysql (DBD::mysql, DBI) on MacBook M1 using Docker