Friday, April 13, 2018

Thursday, April 05, 2018

Thoughts and an example on Expression-Oriented Programming

Here is the code, to read password from .pgpass to connect o a PostgreSQL database.

object DbUtil {
  def dbPassword(hostname:String, port:String, database:String, username:String ):String = {
    // Usage: val thatPassWord = dbPassword(hostname,port,database,username)
    // .pgpass file format, hostname:port:database:username:password
    val passwdFile = new java.io.File(scala.sys.env("HOME"), ".pgpass")
    var passwd = ""
    val fileSrc = scala.io.Source.fromFile(passwdFile)
    fileSrc.getLines.foreach{line =>
      val connCfg = line.split(":")
      if (hostname == connCfg(0)
        && port == connCfg(1)
        && database == connCfg(2)
        && username == connCfg(3)
      ) { 
        passwd = connCfg(4)
      }
    }
    fileSrc.close
    passwd
  }

  def passwordFromConn(connStr:String) = {
    // Usage: passwordFromConn("hostname:port:database:username")
    val connCfg = connStr.split(":")
    dbPassword(connCfg(0),connCfg(1),connCfg(2),connCfg(3))
  }
}

Thoughts.

* foreach v.s.  map().filter()(0) . e.g.: List(1,2,3).filter(_ > 0)(0)
* var passwd, v.s.  val passwd inside of foreach.
* is it good to do chain of method style?

* how to break when find one, in a for loop ?

import util.control.Breaks._
breakable {
for (i <- 1 to 10) {
println(i)
if (i > 4) break // break out of the for loop }
}

* other improvement ?

Wednesday, March 14, 2018

From Python chapter 2 code solution

Here is my answer for book:

从Python开始学编程


Chapter 2, Appendix A, Exam.

https://book.douban.com/subject/26919485/
..

from itertools import repeat

def gen():
int_rate = [0.01, 0.02, 0.03, 0.035, 0.05]
for r in int_rate:
yield r
for r in repeat(0.05):
yield r

def num_year_pay() :
house_price = 500000
year_pay = 30000
year_n = 1
rem = house_price

print(year_pay)

for r in gen():
if rem <=0:
break
print(r)
rem = rem * (1 + r) - year_pay
year_n = year_n + 1
print(rem)

return year_n

if __name__ == "__main__":
print("Number of years to pay loan: ", num_year_pay())

Number of years to pay loan: 31

Thursday, January 11, 2018

work with a custom schema with Spark loading CSV file

Here's how you can work with a custom schema, a complete demo:
Unix shell code,
$>
echo "
Slingo, iOS 
Slingo, Android
" > game.csv
Scala code:
import org.apache.spark.sql.types._

val customSchema = StructType(Array(
  StructField("game_id", StringType, true),
  StructField("os_id", StringType, true)
))

val csv_df = spark.read.format("csv").schema(customSchema).load("game.csv")
csv_df.show 

csv_df.orderBy(asc("game_id"), desc("os_id")).show
csv_df.createOrReplaceTempView("game_view")
val sort_df = sql("select * from game_view order by game_id, os_id desc")
sort_df.show 

Reference,

Tuesday, December 19, 2017

spark 2.2.1 installation guide on Mac

// blog install spark 2.2.1 on Mac

Spark is incompatible with Java 9.
Install Java 8 instead.

# to uninstall Java 9
brew cask uninstall java

Here is the complete and simple installation steps for spark on Mac :

brew tap caskroom/versions
brew cask search java
brew cask install java8

# brew install scala # You might need these step,
brew install apache-spark


# Start spark shell,

./bin/spark-shell --master local[2]

./bin/spark-shell --master local[1]

Scala access PostgresQL database complete demo

// Connect o PostgresQL

$>
brew info postgres

// Start manually:

pg_ctl -D /usr/local/var/postgres start

// or, write stats to log,
pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start

// Stop manually:

pg_ctl -D /usr/local/var/postgres stop

// Status

pg_ctl -D /usr/local/var/postgres status


createdb scala_db
createuser scala_user
psql -U scala_user -d scala_db

SQL>

create table emps
(
id integer,
last_name varchar(100),
email varchar(200),
gender varchar(10),
department varchar(100),
start_date date,
salary integer,
job_title varchar(100),
region_id int,
primary key (id)
);

insert into emps values (1,'Kelley','rkelley0@sours.getString("country")ndcloud.com','Female','Computers','10/2/2009',67470,'Structural Engineer',2);
insert into emps values (2,'Armstrong','sarmstrong1@infoseek.co.jp','Male','Home','3/31/2008',71869,'Financial Advisor',2);
insert into emps values (3,'Carr','fcarr2@woothemes.com','Male','Home','7/12/2009',101768,'Recruiting Manager',3);
insert into emps values (4,'Murray','jmurray3@gov.uk','Female','Jewelery','12/25/2014',96897,'Desktop Support Technician',3);
insert into emps values (5,'Ellis','jellis4@sciencedirect.com','Female','Home','9/19/2002',63702,'Software Engineer III',7);


$>

// scala -classpath ~/app/lib/postgresql-42.1.1.jre6.jar
bin/spark-shell -classpath ~/app/lib/postgresql-42.1.1.jre6.jar

// Scala code,
import java.sql.Driver
import java.sql.DriverManager
import java.sql.Connection

val driver = "org.postgresql.Driver"
val url = "jdbc:postgresql://localhost/scala_db?user=scala_user"
Class.forName(driver)
var connection: Connection = null

connection = DriverManager.getConnection(url)
val statement = connection.createStatement()
val resultSet = statement.executeQuery("select * from emps limit 5")
resultSet.next
resultSet.getString("last_name")
resultSet.getString("department")

while ( resultSet.next() ) {
val last_name = resultSet.getString("last_name")
val dept = resultSet.getString("department")
printf("%s: %s \n", dept, last_name)
}

// Prepare SQL statement
val queryStr = "select * from emps where department = ? "
val ps = connection.prepareStatement(queryStr)
ps.setString(1, "Home")
val rs = ps.executeQuery()

while ( rs.next() ) {
val last_name = rs.getString("last_name")
val dept = rs.getString("department")
val startDate = rs.getString("start_date")
printf("%s: %s \t %s \n", dept, last_name, startDate)
}

Thursday, November 23, 2017

JavaScript Closures: setTimeout and loop

Problem:

for (var i = 0; i < 3; i++) {
    setTimeout(function() {
        console.log("My value: " + i);
    }, 1);
}

My value: 3
My value: 3
My value: 3

}

Why:

Variable Global scope, and JavaScript asynchronous event loop single thread execution.

Solution:

// Block scope, ES2005
for (let i = 0; i < 3; i++) {
    setTimeout(function() {
        console.log("My value: " + i);
    }, 1000);
}

//Immediately Invoked creation, and function scope.
for (var i = 0; i < 3; i++) {
    setTimeout((function(pNum) {
        return function() {
            console.log('i: ', pNum)
        }
    })(i), 1000);
}

//  IIFE (Immediately Invoked Function Expression)
for (var i = 0; i < 3; i++) {
    (function(pNum = i) {
        setTimeout(
            function() {
                console.log('i : ', pNum)
            }, 1000)
    })(i);
}


Enjoy the sunshine,