xsv, ants, resty を試してみた。
k-junです。今週も適当にライブラリを試して行きます。
xsv
rust 製の csv 統計ツール。README.md を見る限りでは使いやすそうなので、適当な csv を生成して操作してみる。 https://github.com/BurntSushi/xsv
ここに csv ファイルがいっぱい転がっているので、ダウンロード。
# 違いは...? ~ xsv cat rows cities.csv LatD," ""LatM"""," ""LatS"""," ""NS"""," ""LonD"""," ""LonM"""," ""LonS"""," ""EW"""," ""City"""," ""State""" 41, 5, 59," ""N""", 80, 39, 0," ""W"""," ""Youngstown""", OH 42, 52, 48," ""N""", 97, 23, 23," ""W"""," ""Yankton""", SD 46, 35, 59," ""N""", 120, 30, 36," ""W"""," ""Yakima""", WA 42, 16, 12," ""N""", 71, 48, 0," ""W"""," ""Worcester""", MA 43, 37, 48," ""N""", 89, 46, 11," ""W"""," ""Wisconsin Dells""", WI 36, 5, 59," ""N""", 80, 15, 0," ""W"""," ""Winston-Salem""", NC 49, 52, 48," ""N""", 97, 9, 0," ""W"""," ""Winnipeg""", MB 39, 11, 23," ""N""", 78, 9, 36," ""W"""," ""Winchester""", VA 34, 14, 24," ""N""", 77, 55, 11," ""W"""," ""Wilmington""", NC ~ xsv cat columns cities.csv | head LatD," ""LatM"""," ""LatS"""," ""NS"""," ""LonD"""," ""LonM"""," ""LonS"""," ""EW"""," ""City"""," ""State""" 41, 5, 59," ""N""", 80, 39, 0," ""W"""," ""Youngstown""", OH 42, 52, 48," ""N""", 97, 23, 23," ""W"""," ""Yankton""", SD 46, 35, 59," ""N""", 120, 30, 36," ""W"""," ""Yakima""", WA 42, 16, 12," ""N""", 71, 48, 0," ""W"""," ""Worcester""", MA 43, 37, 48," ""N""", 89, 46, 11," ""W"""," ""Wisconsin Dells""", WI 36, 5, 59," ""N""", 80, 15, 0," ""W"""," ""Winston-Salem""", NC 49, 52, 48," ""N""", 97, 9, 0," ""W"""," ""Winnipeg""", MB 39, 11, 23," ""N""", 78, 9, 36," ""W"""," ""Winchester""", VA 34, 14, 24," ""N""", 77, 55, 11," ""W"""," ""Wilmington""", NC
うーん。出力が美しくないし、差分もわからん。
~ xsv stats cities.csv field,type,sum,min,max,min_length,max_length,mean,stddev LatD,Unicode,, 26, 50,5,5,, " ""LatM""",Unicode,, 1, 59,5,5,, " ""LatS""",Unicode,, 0, 59,5,5,, " ""NS""",Unicode,," ""N"""," ""N""",4,4,, " ""LonD""",Unicode,, 71, 123,7,7,, " ""LonM""",Unicode,, 0, 58,5,5,, " ""LonS""",Unicode,, 0, 59,5,5,, " ""EW""",Unicode,," ""W"""," ""W""",4,4,, " ""City""",Unicode,," ""Ravenna"""," ""Youngstown""",7,21,, " ""State""",Unicode,, AL, WY,3,4,,
どれもこれも、""で囲まれ過ぎている出力がいけていないせいで、全く便利じゃない。もとのファイルがダブルクオーテーションで囲まているのがいけてない。 もう使わない。
~ curl -LO https://burntsushi.net/stuff/worldcitiespop.csv ~ xsv stats worldcitiespop.csv field,type,sum,min,max,min_length,max_length,mean,stddev Country,Unicode,,ad,zw,2,2,, City,Unicode,, bab el ahmar,Þykkvibaer,1,91,, AccentCity,Unicode,, Bâb el Ahmar,ïn Bou Chella,1,91,, Region,Unicode,,00,Z9,0,2,, Population,Integer,2289584999,7,31480498,0,8,47719.570633597126,302885.5592040396 Latitude,Float,86294096.37312101,-54.933333,82.483333,1,12,27.188165808468785,21.95261384912504 Longitude,Float,117718483.57958724,-179.9833333,180,1,14,37.08885989656418,63.223010459241635
イケてる。最高。あと使いそうなのは、table ぐらいか。
~ xsv table worldcitiespop.csv | head Country City AccentCity Region Population Latitude Longitude ad aixas Aixàs 06 42.4833333 1.4666667 ad aixirivali Aixirivali 06 42.4666667 1.5 ad aixirivall Aixirivall 06 42.4666667 1.5 ad aixirvall Aixirvall 06 42.4666667 1.5 ad aixovall Aixovall 06 42.4666667 1.4833333 ad andorra Andorra 07 42.5 1.5166667 ad andorra la vella Andorra la Vella 07 20430 42.5 1.5166667 ad andorra-vieille Andorra-Vieille 07 42.5 1.5166667 ad andorre Andorre 07 42.5 1.5166667
ants
goroutine の pooling 管理を行うツール。goroutine を pool することで並列に走らせる際に、その数を制限したりできる。 内部で semaphone とか使用してそう。分散系のシステムを構築する際には、便利になるかもしれない。
https://github.com/panjf2000/ants
README.md を参考に試してみる。
https://github.com/panjf2000/antspackage main import ( "fmt" "sync" "sync/atomic" "time" "github.com/panjf2000/ants/v2" ) var sum int32 func myFunc(i interface{}) { n := i.(int32) atomic.AddInt32(&sum, n) } func demoFunc() { time.Sleep(10 * time.Millisecond) } func main() { defer ants.Release() runTimes := 1000 // Use the common pool. var wg sync.WaitGroup p, _ := ants.NewPoolWithFunc(10, func(i interface{}) { myFunc(i) wg.Done() }) defer p.Release() // Submit tasks one by one. for i := 0; i < runTimes; i++ { wg.Add(1) _ = p.Invoke(int32(i)) fmt.Printf("running goroutines: %d\n", p.Running()) } wg.Wait() fmt.Printf("finish all tasks, result is %d\n", sum) }
~/source-code/playground-golang(master ✗) go run ./main.go | head -n 20 running goroutines: 1 running goroutines: 2 running goroutines: 3 running goroutines: 4 running goroutines: 5 running goroutines: 6 running goroutines: 7 running goroutines: 8 running goroutines: 9 running goroutines: 10 running goroutines: 10 running goroutines: 10 running goroutines: 10 running goroutines: 10 running goroutines: 10 running goroutines: 10 running goroutines: 10 running goroutines: 10 running goroutines: 10 running goroutines: 10
いい感じ。規模が大きくなると、速度とメモリの観点から利点が出てくるらしい。
https://github.com/panjf2000/ants#-performance-summary
resty
golang で書かれたシンプルな http client ツール。
https://github.com/go-resty/resty
使ってみた限りはシンプルで全然使えそう。post の際にも自動的に "Content-Type" 入れてくれたし。
package main import ( "fmt" "github.com/go-resty/resty/v2" ) func main() { // html client := resty.New() resp, _ := client.R().Get("https://guthib.com") fmt.Println(string(resp.Body())) // json get resp, _ = client.R().Get("https://httpbin.org/get") fmt.Println(string(resp.Body())) // json post resp, _ = client.R().SetBody(`{"key": "value"}`).Post("https://httpbin.org/post") fmt.Println(string(resp.Body())) }
<style type="text/css"> h1 { text-align: center; font-size: 120px; font-family: Helvetica, Verdana, Arial; } </style> <h1>You spelled it wrong.</h1> { "args": {}, "headers": { "Accept-Encoding": "gzip", "Host": "httpbin.org", "User-Agent": "go-resty/2.6.0 (https://github.com/go-resty/resty)", "X-Amzn-Trace-Id": "Root=1-60e97827-01676b76592fe46543086c25" }, "origin": "106.165.44.175", "url": "https://httpbin.org/get" } { "args": {}, "data": "{\"key\": \"value\"}", "files": {}, "form": {}, "headers": { "Accept-Encoding": "gzip", "Content-Length": "16", "Content-Type": "text/plain; charset=utf-8", "Host": "httpbin.org", "User-Agent": "go-resty/2.6.0 (https://github.com/go-resty/resty)", "X-Amzn-Trace-Id": "Root=1-60e97828-3b544cf11b6f9c6e7e892286" }, "json": { "key": "value" }, "origin": "106.165.44.175", "url": "https://httpbin.org/post" }
ではでは今日はこのへんで。
