<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Backgrounds
I just got rejected by a big tech company. **In order to** improve my ability to explain interview questions effectively in English, I'm designing an interview warmup tool using LLM and RAG technologies.
## Design
So, I start design a system that could help me practice:
- User Story
- As a recruiter, I want to disguise the user’s answer is correct or not based on the QA dataset so that I could filter potential candidates.
- Scope the Problem
- Recruiter can build his/her QA datasets.
- LLM will provide some technical advice to the recruiter
- Main component
- Storage: JSON file
- Advice: ChatGPT+RAG
![](https://i.imgur.com/n7v6IgB.png)
This project consists of two main components: Retriever and Advisor. Retriever focuses on managing the dataset. Initially, I use Gemini to generate the correct answers, which are then saved in a JSON file. Subsequently, questions and answers are stored in a JSON file due to its advantages:
- **Machine-readable:** Computers can easily process the data.
- **Human-readable:** Users can understand the content readily.
- **Consolidated storage:** All questions and answers are conveniently located in a single file.
To find relevant questions quickly, Retriever employs embedding techniques.
The Advisor component, powered by the ChatGPT API, assesses candidate responses **to** the recruiter's questions. Advisor will then provide its results to the recruiter.
## Sample Code
- `qa.json`
```python
{
"tq":[
{
"q":"What is the order of elements return by the keys() method of a python dict?",
"a":"The order of elements returned by the keys() method of a Python dictionary is not guaranteed to be in any specific order. It may reflect the order the items were added in some cases, but this behavior is not reliable and can change between Python versions. If you depend on a specific order for your keys, you can use the sorted() function along with the keys() method."
}
]
}
```
- main.py
```python
import json
from pathlib import Path
file_path='qa.json'
data = json.loads(Path(file_path).read_text())
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.prompts import ChatPromptTemplate
embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
db = Chroma.from_documents(documents, embedding_function)
retriever = db.as_retriever()
template = """Please distinguish the answer is correct or not based only on the following context and question, if the answer is incorrect, please explain why answer is incorrect.:
{context}
Question: {question}
Answer: {answer}
"""
prompt = ChatPromptTemplate.from_template(template)
from langchain_core.runnables import RunnableLambda, RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
import os
os.environ['OPENAI_API_KEY'] = "sk-OOOO"
model = ChatOpenAI()
question = "what is the return order of the dict by using keys() method"
chain = (
{"context": retriever, "answer": RunnablePassthrough(), "question": RunnablePassthrough()}
| prompt
| model
| StrOutputParser()
)
query = "I'm not sure the order. But If I thought I will first sorted by its key to granted the return order of the elements"
print(chain.invoke(query))
```
## Notes
- https://python.langchain.com/docs/get_started/installation

</body>
</html>


## Backgrounds
I just got rejected by a big tech company. **In order to** improve my ability to explain interview questions effectively in English, I'm designing an interview warmup tool using LLM and RAG technologies.
## Design
So, I start design a system that could help me practice:
- User Story
	- As a recruiter, I want to disguise the user’s answer is correct or not based on the QA dataset so that I could filter potential candidates.
- Scope the Problem
	- Recruiter can build his/her QA datasets.
	- LLM will provide some technical advice to the recruiter
- Main component
	- Storage: JSON file
	- Advice: ChatGPT+RAG
![](https://i.imgur.com/n7v6IgB.png)

This project consists of two main components: Retriever and Advisor. Retriever focuses on managing the dataset. Initially, I use Gemini to generate the correct answers, which are then saved in a JSON file. Subsequently, questions and answers are stored in a JSON file due to its advantages:
- **Machine-readable:** Computers can easily process the data.
- **Human-readable:** Users can understand the content readily.
- **Consolidated storage:** All questions and answers are conveniently located in a single file.
To find relevant questions quickly, Retriever employs embedding techniques.
The Advisor component, powered by the ChatGPT API, assesses candidate responses **to** the recruiter's questions. Advisor will then provide its results to the recruiter.  

## Sample Code
- `qa.json`
```python
{
   "tq":[
      {
         "q":"What is the order of elements return by the keys() method of a python dict?",
         "a":"The order of elements returned by the keys() method of a Python dictionary is not guaranteed to be in any specific order. It may reflect the order the items were added in some cases, but this behavior is not reliable and can change between Python versions. If you depend on a specific order for your keys, you can use the sorted() function along with the keys() method."
      }
   ]
}
```
- main.py
```python
import json
from pathlib import Path
file_path='qa.json'
data = json.loads(Path(file_path).read_text())

from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.prompts import ChatPromptTemplate
embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
db = Chroma.from_documents(documents, embedding_function)
retriever = db.as_retriever()
template = """Please distinguish the answer is correct or not based only on the following context and question, if the answer is incorrect, please explain why answer is incorrect.:
{context}
Question: {question}
Answer: {answer}
"""
prompt = ChatPromptTemplate.from_template(template)
from langchain_core.runnables import RunnableLambda, RunnablePassthrough

from langchain_core.output_parsers import StrOutputParser

from langchain_openai import ChatOpenAI

import os

os.environ['OPENAI_API_KEY'] = "sk-OOOO"

model = ChatOpenAI()

question = "what is the return order of the dict by using keys() method"

chain = (

{"context": retriever, "answer": RunnablePassthrough(), "question": RunnablePassthrough()}
| prompt
| model
| StrOutputParser()
)

query = "I'm not sure the order. But If I thought I will first sorted by its key to granted the return order of the elements"
print(chain.invoke(query))
```
## Notes
- https://python.langchain.com/docs/get_started/installation

20240321-interview-warmup

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

使用 AES（對稱加密） + base64
## 加密
```python
from Crypto.Cipher import AES
from Crypto.Util.Padding import pad, unpad
from Crypto.Protocol.KDF import PBKDF2
from Crypto.Random import get_random_bytes
salt = get_random_bytes(16)
key_enc = PBKDF2(password, salt, dkLen=32)
cipher = AES.new(key_enc, AES.MODE_ECB)
ciphertext = cipher.encrypt(pad(plaintext.encode(), BLOCK_SIZE))
data[k] = base64.b64encode(ciphertext).decode()
with open("password.txt", "w") as f:
json.dump(data, f)
```
## Ref

</body>
</html>


使用 AES（對稱加密） + base64 
## 加密
```python
from Crypto.Cipher import AES
from Crypto.Util.Padding import pad, unpad
from Crypto.Protocol.KDF import PBKDF2
from Crypto.Random import get_random_bytes

salt = get_random_bytes(16)
key_enc = PBKDF2(password, salt, dkLen=32)
cipher = AES.new(key_enc, AES.MODE_ECB)
ciphertext = cipher.encrypt(pad(plaintext.encode(), BLOCK_SIZE))
data[k] = base64.b64encode(ciphertext).decode()
with open("password.txt", "w") as f:
	json.dump(data, f)
```

## Ref


20240303-save-password-python

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

發現一個有趣的東西
brew install --cask warp
![](https://assets-global.website-files.com/64b6f3636f598299028e8577/65d5db665ae2b6b691e54ac6_modern-editing%402x.webp)
## Ref
-

</body>
</html>


發現一個有趣的東西
brew install --cask warp
![](https://assets-global.website-files.com/64b6f3636f598299028e8577/65d5db665ae2b6b691e54ac6_modern-editing%402x.webp)
## Ref
- 


20240227-warp-terminal

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
設計一個使用chaining（Link List）處理碰撞問題的hash table
## Solution
O(1) 可以使用array但浪費空間
可以使用binary search但就不會是O(1)
## Ref

</body>
</html>


## Question
設計一個使用chaining（Link List）處理碰撞問題的hash table
## Solution
O(1) 可以使用array但浪費空間
可以使用binary search但就不會是O(1)
## Ref


20240227-Hash-Table-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
解釋如何設計in-memory的DS 及 演算法，請可能舉例
## Solution
最簡單就分成有File及Directory他們各自extend Entry
## Ref

</body>
</html>


## Question
解釋如何設計in-memory的DS 及 演算法，請可能舉例
## Solution
最簡單就分成有File及Directory他們各自extend Entry
## Ref


20240227-File-Systems-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
踩地雷遊戲，玩家直到整個盤都探索完畢，玩家點選後如果是炸彈就輸掉，若是空白，消除所有鄰近空白及數字
## Solution
- focus key idea+structure out
### Design:Cell
- enum
### Design:Board
- init, flip
### Design: Game
- host game state
### Algorithm
#### placing the bomb
#### setting number cells
#### expanding a Blank Region
## Ref

</body>
</html>


## Question
踩地雷遊戲，玩家直到整個盤都探索完畢，玩家點選後如果是炸彈就輸掉，若是空白，消除所有鄰近空白及數字
## Solution
- focus key idea+structure out
### Design:Cell
- enum
### Design:Board
- init, flip
### Design: Game
- host game state
### Algorithm
#### placing the bomb
#### setting number cells
#### expanding a Blank Region


## Ref


20240226-minesweeper-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
實作CircularArray class可以有效翻轉array like data structure，這個class需使用generic type(template)，且支援iter notaion（obj o: circularArray）
## Solution
- 拆解問題：實作CircularArray class、支援iteration
### Implementing the CircularArray class
- 注意負數在mod後的使用
### Implementing the Iterator Interface
這邊參考p330
## Ref

</body>
</html>


## Question
實作CircularArray class可以有效翻轉array like  data structure，這個class需使用generic type(template)，且支援iter notaion（obj o: circularArray）
## Solution
- 拆解問題：實作CircularArray class、支援iteration
### Implementing the CircularArray class
- 注意負數在mod後的使用
### Implementing the Iterator Interface
這邊參考p330


## Ref


20240226-Circular-array-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

- 文字清理（去除符號、轉小寫）
- 依據詞頻拆分關鍵字網路
- 邊/連結厚度代表關鍵字的相依頻率
- 節點大小代表專見字的頻率
## Ref

</body>
</html>


- 文字清理（去除符號、轉小寫）
- 依據詞頻拆分關鍵字網路
- 邊/連結厚度代表關鍵字的相依頻率
- 節點大小代表專見字的頻率
## Ref


20240222-content-analysis

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
- 設計黑白棋的OOD，黑白棋是如果周遭（上下左右）被圍起來就更換中間（被包圍）顏色，到一方無法下任何棋結束。
## Solution
- core object: game, board, pieces, players
### Shold BlackPiece and WhitePiece be classes?
- 因為棋會一直被翻轉，有一個統一Piece Class可能比較好
### Do we need separate Board and Game classes?
- Pros
- 邏輯上分開board跟game
- Cons
- 需要額外layer
## Who keeps Score?
- 可能可以使用Group、Piece、Board group
- 我們暫時使用board存，這樣分數可以由Board分組
## Should Game be a Singleton class?
- Singleton 好處是所有人可呼叫，而不用每次都pass ref，但只能被初始化一次，與interviewer討論是否可以這樣做
## Ref

</body>
</html>


## Question
- 設計黑白棋的OOD，黑白棋是如果周遭（上下左右）被圍起來就更換中間（被包圍）顏色，到一方無法下任何棋結束。
## Solution
- core object: game, board, pieces, players
### Shold BlackPiece and WhitePiece be classes?
- 因為棋會一直被翻轉，有一個統一Piece Class可能比較好
### Do we need separate Board and Game classes?
- Pros
	- 邏輯上分開board跟game
- Cons
	- 需要額外layer
## Who keeps Score?
- 可能可以使用Group、Piece、Board group
- 我們暫時使用board存，這樣分數可以由Board分組
## Should Game be a Singleton class?
- Singleton 好處是所有人可呼叫，而不用每次都pass ref，但只能被初始化一次，與interviewer討論是否可以這樣做
## Ref


20240221-othello-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
- 請解釋你會怎麼設計一個chat server，請說明你會怎麼設計Backend Component的細節、Class及Method，可能需要解決最困難的問題是什麼
## Solution
- 這問題很大，一開始要scope problem，一開始建議從廣下手，但focus可以再inerview完成的部分
- focus 使用者管理（新增、建立對話、更新狀態）
- 先不管網路、資料怎麼傳送給client等問題
- 要加入聯絡人，要兩方都是才是，支援群組聊天、一對一私人對話，不考慮語、音通話或檔案傳輸
### What specific action does it need to support?
- 與interviewer討論，for example:
- singing online and offline
- 加入request(sending, accepting and rejeccting)
- 更新狀態
- 建立private 及群組訊息
- 在private及群組對話加入訊息
### What can we learn about these requirements?
- 我們需要有使用者、新增request狀態、線上狀態、訊息
### What are the core components of the system?
- 使用database永久儲存資料，SQL很好，如果要scalability，可以使用BigTable之類
- client-server溝通可以使用XML，因為人跟機器可讀，但他不是最好壓縮過的格式
- 資料可能被切分在不同機器，為了避免single point of failure，資料會複製好幾份在不同機器
### What are the key objects and methods
- key objects, includes users, conversations, status message
- UserManager
- userById: map(int=>User)
- userByAccountName: map(string=>User)
- onlineUsers: map(int=>User)
- addUser()
- approveAddRequest()
- rejectAddRequest()
- iserSignedOn()
- iserSignedOff()
### What problems would be the hardest to solve(or the most interesting)?
- 可能會interviewer討論以下問題
- Q1: How do we know If someone is online--I mean, really know?
- 有可能使用者忘記登出，定時去ping client看是否還在
- Q2: How do we deal with conflicting information
- 有些存databse，有些存電腦的memory，如果有些out of sync怎麼辦？哪個才正確？
- Q3: How do we make our server scale?
- 如何設計可以擴展的系統，真實世界資料可能要拆分在不同機器，怎麼處理不同步資料
## Ref

</body>
</html>


## Question
- 請解釋你會怎麼設計一個chat server，請說明你會怎麼設計Backend Component的細節、Class及Method，可能需要解決最困難的問題是什麼
## Solution
- 這問題很大，一開始要scope problem，一開始建議從廣下手，但focus可以再inerview完成的部分
- focus 使用者管理（新增、建立對話、更新狀態）
- 先不管網路、資料怎麼傳送給client等問題
- 要加入聯絡人，要兩方都是才是，支援群組聊天、一對一私人對話，不考慮語、音通話或檔案傳輸
### What specific action does it need to support?
- 與interviewer討論，for example:
	- singing online and offline
	- 加入request(sending, accepting and rejeccting)
	- 更新狀態
	- 建立private 及群組訊息
	- 在private及群組對話加入訊息
### What can we learn about these requirements?
- 我們需要有使用者、新增request狀態、線上狀態、訊息
### What are the core components of the system?
- 使用database永久儲存資料，SQL很好，如果要scalability，可以使用BigTable之類
- client-server溝通可以使用XML，因為人跟機器可讀，但他不是最好壓縮過的格式
- 資料可能被切分在不同機器，為了避免single point of failure，資料會複製好幾份在不同機器
### What are the key objects and methods
- key objects, includes users, conversations, status message
	- UserManager
		- userById: map(int=>User)
		- userByAccountName: map(string=>User)
		- onlineUsers: map(int=>User)
		- addUser()
		- approveAddRequest()
		- rejectAddRequest()
		- iserSignedOn() 
		- iserSignedOff()
### What problems would be the hardest to solve(or the most interesting)?
- 可能會interviewer討論以下問題
- Q1: How do we know If someone is online--I mean, really know?
	- 有可能使用者忘記登出，定時去ping client看是否還在
- Q2: How do we deal with conflicting information
	- 有些存databse，有些存電腦的memory，如果有些out of sync怎麼辦？哪個才正確？
- Q3: How do we make our server scale?
	- 如何設計可以擴展的系統，真實世界資料可能要拆分在不同機器，怎麼處理不同步資料



## Ref


20240219-chat-server-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
設計一個根據object-oriented principles的parking lot（停車場）
## Solution
- 跟interviewer討論支援什麼類別的車
- 假設
- 停車場有多個level，每個level有多個spots
- 停車長可以停摩托車、汽車及公車
- 有摩托車停車、小客車停車場及大型停車場
- 摩托車可以停在任何停車場
- 小客車可以停在小客車、大型停車場
- 公車只能停在大型停車場
- 建立一個車輛的abstract類別可以讓其他extend
- ParkingLot是多個Level的warpper
- 摩托車停車、小客車停車場及大型停車場可以繼承自ParkingLot
## Ref

</body>
</html>


## Question
設計一個根據object-oriented principles的parking lot（停車場）
## Solution
- 跟interviewer討論支援什麼類別的車
- 假設
	- 停車場有多個level，每個level有多個spots
	- 停車長可以停摩托車、汽車及公車
	- 有摩托車停車、小客車停車場及大型停車場
	- 摩托車可以停在任何停車場
	- 小客車可以停在小客車、大型停車場
	- 公車只能停在大型停車場
- 建立一個車輛的abstract類別可以讓其他extend
- ParkingLot是多個Level的warpper
- 摩托車停車、小客車停車場及大型停車場可以繼承自ParkingLot

## Ref


20240217-parking-lot-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
設計線上書本閱讀器的資料結構
## Solution
- 因為問題沒解釋很清楚，我們假設提供以下功能
- 用戶帳戶建立、擴增
- 搜尋資料庫的書
- 看書
- 同一時間只可以有一個線上使用者
- 這個使用者指只會有一本acitve的書
- 有可能會有
- User/Book/Library
- get/set/update
## Ref

</body>
</html>


## Question
設計線上書本閱讀器的資料結構
## Solution
- 因為問題沒解釋很清楚，我們假設提供以下功能
	- 用戶帳戶建立、擴增
	- 搜尋資料庫的書
	- 看書
	- 同一時間只可以有一個線上使用者
	- 這個使用者指只會有一本acitve的書
- 有可能會有
	- User/Book/Library
	- get/set/update

## Ref


20240217-online-book-reader-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
使用object-oriented principles設計jukebox musical（點唱機）
## Solution
- 問interviewer釐清設計限制
- 播放CD/錄影/MP3？
- 在PC/實體點唱機？
- 假設
- for example 電腦模擬，要接近實體點唱機
- basic component
- Jukebox
- CD
- Song
- Artist
- Playlist
- Display
- breakdown to find possible actions
- Playlist creation（add/delete/shuffle)
- CD selector
- Song selector
- Queuing up a song
- get next song from playlist
- 使用者可以：
- add
- delete
- credit infomation
-
## Ref

</body>
</html>


## Question
使用object-oriented principles設計jukebox musical（點唱機）
## Solution
- 問interviewer釐清設計限制
	- 播放CD/錄影/MP3？
	- 在PC/實體點唱機？
- 假設
	- for example 電腦模擬，要接近實體點唱機
- basic component
	- Jukebox
	- CD
	- Song
	- Artist
	- Playlist
	- Display
- breakdown to find possible actions
	- Playlist creation（add/delete/shuffle)
	- CD selector
	- Song selector
	- Queuing up a song
	- get next song from playlist
- 使用者可以：
	- add
	- delete
	- credit infomation
- 
## Ref


20240217-jukebox-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
打造一個拼圖遊戲，設計資料結構以及解釋如何破解這個拼圖的演算法。你可以假設有一個fitsWith()，他可以檢查兩個拼圖是否可以拼在一起。
## Solution
- 我們需要針對每一片拼圖儲存位置，可能有兩種儲存情況
- 絕對位置：(x,y)
- 相對位置：還不知道在哪，但知道A在B旁邊
- 主要Class: Puzzle, Piece, Edge
- 形狀(shape): inner, outer, flat
- edge的方向: left, right, top, down
- Puzzle 會有個很多piece的list
- Piece 會有個hashtable(orientation=>edge)，有可能會rotate piece，所以hash有可能會變
### Algorithm to Solve the Puzzle
- 要可以rotate嘗試是否可以塞到角落
## Ref

</body>
</html>


## Question
打造一個拼圖遊戲，設計資料結構以及解釋如何破解這個拼圖的演算法。你可以假設有一個fitsWith()，他可以檢查兩個拼圖是否可以拼在一起。

## Solution
- 我們需要針對每一片拼圖儲存位置，可能有兩種儲存情況
	- 絕對位置：(x,y)
	- 相對位置：還不知道在哪，但知道A在B旁邊
- 主要Class: Puzzle, Piece, Edge
- 形狀(shape): inner, outer, flat
- edge的方向: left, right, top, down
- Puzzle 會有個很多piece的list
- Piece 會有個hashtable(orientation=>edge)，有可能會rotate piece，所以hash有可能會變
### Algorithm to Solve the Puzzle
- 要可以rotate嘗試是否可以塞到角落
## Ref


20240217-jigsaw-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
有一個有三種level員工的call center，respondent, manager, director。當電話進來時，要assign給空閒的respondent，如果他不能回答，必須轉介給manager，也不行就往上給director，設計一個給這問題的class及資料結構，實作一個dispathchCall()指派電話給低一個可用的員工
## solution
- OOD有很多方式可以實作，跟面試官討論，trade off，保持彈性、可維護性設計。
- Call代表一則通話，並且call有最小的rank而且被指定給可以處理的第一個員工
- Employee 是superclass，且是一個abstract類別，不應該直接實例化
- respondent, manager, director可以extend Employee類別
## Ref

</body>
</html>


## Question
有一個有三種level員工的call center，respondent, manager, director。當電話進來時，要assign給空閒的respondent，如果他不能回答，必須轉介給manager，也不行就往上給director，設計一個給這問題的class及資料結構，實作一個dispathchCall()指派電話給低一個可用的員工
## solution
- OOD有很多方式可以實作，跟面試官討論，trade off，保持彈性、可維護性設計。
- Call代表一則通話，並且call有最小的rank而且被指定給可以處理的第一個員工
- Employee 是superclass，且是一個abstract類別，不應該直接實例化
- respondent, manager, director可以extend Employee類別
## Ref


20240217-call-center-ood

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Background
- from a [stack overflow question](https://stackoverflow.com/questions/76496884/how-levels-fyi-is-encoding-the-api-response) , level.fyi encrypt its response
- request
```javascript
GET https://api.levels.fyi/v3/salary/search?countryIds[]=197&#x26;offset=10&#x26;limit=50&#x26;sortBy=offer_date&#x26;sortOrder=DESC&#x26;jobFamilySlug=software-engineer
```
- response
```json
{
"payload": "EofXi7jF2t63a..."
}
```
## solution
- Go to [levels.fyi](https://www.levels.fyi/)> Inspect>Network>JS tab>commonUtils.js>CryptoJS.AES.decrypt
- I wrote a python script :
```python
from Crypto.Cipher import AES
from Crypto.Hash import MD5
from base64 import b64encode, b64decode
import zlib
class ResponseUtil:
def __init__(self):
self.key = "levelstothemoon!!"
self.n = 16
def parse(self, t):
if "payload" not in t:
return t
r = t["payload"]
a = MD5.new(self.key.encode()).digest()
a_base64 = b64encode(a)[: self.n]
cipher = AES.new(a_base64, AES.MODE_ECB)
decrypted_data = cipher.decrypt(b64decode(r))
decompressed_data = zlib.decompress(decrxypted_data)
return json.loads(decompressed_data.decode())
# Example usage:
response_util = ResponseUtil()
parsed_data = response_util.parse(ans)
print(parsed_data)
```
```json
{
"total": 1000,
"hidden": 2,
"rows": [
{
"uuid": "079fb0cf-d9ff-4b58-bb08-70f3a8447521",
"title": "Software Engineer",
"jobFamily": "Software Engineer",
"level": "E7",
"focusTag": "General",
"yearsOfExperience": 5,
"yearsAtCompany": 2,
"yearsAtLevel": 2,
"offerDate": "2024-01-27T02:02:19.682Z",
"location": "Hsin-chu, TP, Taiwan",
"workArrangement": "office",
"compPerspective": "employee",
"cityId": 17410,
"dmaId": 10064,
"countryId": 236,
"exchangeRate": 31.2881,
"baseSalary": 42188.5583,
"baseSalaryCurrency": "TWD",
"totalCompensation": 67757.3815,
"avgAnnualStockGrantValue": null,
"stockGrantCurrency": null,
"avgAnnualBonusValue": 25568.8232,
"bonusCurrency": "TWD",
"salesComp": null,
"negotiatedAmount": null,
"gender": null,
"ethnicity": null,
"education": null,
"otherDetails": null,
"companyInfo": {
"registered": true,
"icon": "https://logo.clearbit.com/mediatek.com",
"name": "MediaTek",
"slug": "mediatek"
},
"vestingSchedule": null,
"tags": null,
"stockType": null
},
...
```
- https://hackmd.io/DXNwbKGJRWamrYYFv6LT_Q
## contract
If you have any questions or needs, please contract me at alan.tseng.cs@gmail.com

</body>
</html>


## Background
- from a [stack overflow question](https://stackoverflow.com/questions/76496884/how-levels-fyi-is-encoding-the-api-response) , level.fyi encrypt its response
- request
```javascript
GET https://api.levels.fyi/v3/salary/search?countryIds[]=197&offset=10&limit=50&sortBy=offer_date&sortOrder=DESC&jobFamilySlug=software-engineer
```
- response
```json
{
    "payload": "EofXi7jF2t63a..."
}
```
## solution
- Go to [levels.fyi](https://www.levels.fyi/)> Inspect>Network>JS tab>commonUtils.js>CryptoJS.AES.decrypt
- I wrote a python script : 

```python
from Crypto.Cipher import AES
from Crypto.Hash import MD5
from base64 import b64encode, b64decode
import zlib
class ResponseUtil:
    def __init__(self):
        self.key = "levelstothemoon!!"
        self.n = 16

    def parse(self, t):
        if "payload" not in t:
            return t
        r = t["payload"]
        a = MD5.new(self.key.encode()).digest()
        a_base64 = b64encode(a)[: self.n]
        cipher = AES.new(a_base64, AES.MODE_ECB)

        decrypted_data = cipher.decrypt(b64decode(r))
        
        decompressed_data = zlib.decompress(decrxypted_data)

        return json.loads(decompressed_data.decode())

# Example usage:
response_util = ResponseUtil()
parsed_data = response_util.parse(ans)
print(parsed_data)

```

```json
{
    "total": 1000,
    "hidden": 2,
    "rows": [
        {
            "uuid": "079fb0cf-d9ff-4b58-bb08-70f3a8447521",
            "title": "Software Engineer",
            "jobFamily": "Software Engineer",
            "level": "E7",
            "focusTag": "General",
            "yearsOfExperience": 5,
            "yearsAtCompany": 2,
            "yearsAtLevel": 2,
            "offerDate": "2024-01-27T02:02:19.682Z",
            "location": "Hsin-chu, TP, Taiwan",
            "workArrangement": "office",
            "compPerspective": "employee",
            "cityId": 17410,
            "dmaId": 10064,
            "countryId": 236,
            "exchangeRate": 31.2881,
            "baseSalary": 42188.5583,
            "baseSalaryCurrency": "TWD",
            "totalCompensation": 67757.3815,
            "avgAnnualStockGrantValue": null,
            "stockGrantCurrency": null,
            "avgAnnualBonusValue": 25568.8232,
            "bonusCurrency": "TWD",
            "salesComp": null,
            "negotiatedAmount": null,
            "gender": null,
            "ethnicity": null,
            "education": null,
            "otherDetails": null,
            "companyInfo": {
                "registered": true,
                "icon": "https://logo.clearbit.com/mediatek.com",
                "name": "MediaTek",
                "slug": "mediatek"
            },
            "vestingSchedule": null,
            "tags": null,
            "stockType": null
        },
        ...
```

- https://hackmd.io/DXNwbKGJRWamrYYFv6LT_Q

## contract
If you have any questions or needs, please contract me at alan.tseng.cs@gmail.com

20240216-levelsfyi-crawler

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
Deck of cards：設計一個給一副牌的generic資料結構，解釋如何設計black-jack(21點)資料結構的子類別
## Solution
- 首先跟面試關確認generic的定義，給Uno、pocker-like還是？
- 假設是poker card:
- Suit
- club, diamond, heart, spade
- value
- getValue()
- getSuitFromValue()
- Deck
- `cards=[]`
- shuffle()
- remainintCards()
- dealHand()
- dealCard()
- Card
- faceValue
- suit
- isAvaliable()
- setAvaliable(state)
- Hand
- `cards=[]`
- score()
- addCard()
- BlackJackHand extends Hand
備註：這邊我除了看書內容，可以問bard: 21點遊戲規則是什麼，請解釋並說明如何使用OOP方式實作
## Ref

</body>
</html>


## Question
Deck of cards：設計一個給一副牌的generic資料結構，解釋如何設計black-jack(21點)資料結構的子類別

## Solution
- 首先跟面試關確認generic的定義，給Uno、pocker-like還是？
- 假設是poker card:
	- Suit
		- club, diamond, heart, spade
		- value
		- getValue()
		- getSuitFromValue()
	- Deck
		- `cards=[]`
		- shuffle()
		- remainintCards()
		- dealHand()
		- dealCard()
	- Card
		- faceValue
		- suit
		- isAvaliable()
		- setAvaliable(state)
	- Hand
		- `cards=[]`
		- score()
		- addCard()
	- BlackJackHand extends Hand

備註：這邊我除了看書內容，可以問bard: 21點遊戲規則是什麼，請解釋並說明如何使用OOP方式實作
## Ref


20240205-Deck-of-cards

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

思路：先用playwright 取得cookies，觀察network直接打API
先取得 geoid (地點id)
```python
q_keywords = "software engineer"
q_location = "taiwan"
r = requests.get(f"https://www.linkedin.com/jobs/search?keywords={q_keywords}&#x26;location={q_location}&#x26;trk=public_jobs_jobs-search-bar_search-submit")
soup = BeautifulSoup(r.text, "html.parser")
geo_id_inputs = soup.select("input[name='geoId']")
geoid = geo_id_inputs[0].get("value")
```
```python
start_n = 0
job_search_link = f"https://www.linkedin.com/voyager/api/voyagerJobsDashJobCards?decorationId=com.linkedin.voyager.dash.deco.jobs.search.JobSearchCardsCollection-192&#x26;count=25&#x26;q=jobSearch&#x26;query=(origin:JOB_SEARCH_PAGE_SEARCH_BUTTON,keywords:software%20engineer,locationUnion:(geoId:{geoid}),spellCorrectionEnabled:true)&#x26;start={start_n}"
headers = [{
'Authority': 'www.linkedin.com',
'Method': 'GET',
'Scheme': 'https',
'Accept': 'application/vnd.linkedin.normalized+json+2.1',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'Cookie': "; ".join([f"{key}={value}" for key, value in session.cookies.items()]),
'Csrf-Token': session.cookies.get('JSESSIONID').strip('"'),
'User-Agent': 'OOOO',
'X-Li-Track': 'OOOO'
} for session in [s]]
r = s.get(job_search_link,headers=headers[0])
```
後記：結果發現github有人做了 [linkedin api](https://github.com/tomquirk/linkedin-api)QQ，當作練習吧
- https://github.com/tomquirk/linkedin-api
-
- 這裡有說明如何運作 https://github.com/tomquirk/linkedin-api?tab=readme-ov-file#how-it-works
```python
from linkedin_api import Linkedin
api = Linkedin('acc', 'pass')
profile = api.get_profile('alanhc316')
contact_info = api.get_profile_contact_info('username')
connections = api.get_profile_connections('username')
```
- [官方API](https://github.com/linkedin-developers/linkedin-api-python-client)
## Ref

</body>
</html>


思路：先用playwright 取得cookies，觀察network直接打API

先取得 geoid (地點id)
```python
q_keywords = "software engineer"
q_location = "taiwan"
r = requests.get(f"https://www.linkedin.com/jobs/search?keywords={q_keywords}&location={q_location}&trk=public_jobs_jobs-search-bar_search-submit")
soup = BeautifulSoup(r.text, "html.parser")
geo_id_inputs = soup.select("input[name='geoId']")
geoid = geo_id_inputs[0].get("value")
```

```python
start_n = 0
job_search_link = f"https://www.linkedin.com/voyager/api/voyagerJobsDashJobCards?decorationId=com.linkedin.voyager.dash.deco.jobs.search.JobSearchCardsCollection-192&count=25&q=jobSearch&query=(origin:JOB_SEARCH_PAGE_SEARCH_BUTTON,keywords:software%20engineer,locationUnion:(geoId:{geoid}),spellCorrectionEnabled:true)&start={start_n}"
headers = [{
'Authority': 'www.linkedin.com',
'Method': 'GET',
'Scheme': 'https',
'Accept': 'application/vnd.linkedin.normalized+json+2.1',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'Cookie': "; ".join([f"{key}={value}" for key, value in session.cookies.items()]),
'Csrf-Token': session.cookies.get('JSESSIONID').strip('"'),
'User-Agent': 'OOOO',
'X-Li-Track': 'OOOO'
} for session in [s]]

r = s.get(job_search_link,headers=headers[0])
```
後記：結果發現github有人做了 [linkedin api](https://github.com/tomquirk/linkedin-api)QQ，當作練習吧
	- https://github.com/tomquirk/linkedin-api
	- 
- 這裡有說明如何運作 https://github.com/tomquirk/linkedin-api?tab=readme-ov-file#how-it-works
```python
from linkedin_api import Linkedin

api = Linkedin('acc', 'pass')

profile = api.get_profile('alanhc316')

contact_info = api.get_profile_contact_info('username')

connections = api.get_profile_connections('username')
```
- [官方API](https://github.com/linkedin-developers/linkedin-api-python-client)
## Ref


20240212-linkedin-private-api

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

[網頁互動版](https://alanhc.github.io/taiwan_jobs.html)
- 資料來源：Levels Fyi
## 台灣
### 在台灣 軟體工程師 做什麼最賺錢？
![](https://i.imgur.com/GHvjK50.png)
Android先猜應該是因為Google的關係拉高平均，Testing有點意外，Backend也是常常被提到成長幅度較大的，Networking先猜應該是網通廠MTK之類的。
AI 天花板高一點，但整體比做產品、嵌入式系統低。
### 在台灣 軟體工程師 做什麼底薪最好？
![](https://i.imgur.com/TnOjosz.png)
起薪最高Android，但成長幅度最高是Backend，但只有很少一部分
### 在台灣 Android 去哪間？
![](https://i.imgur.com/9hAIRym.png)
果然Google最高，但起薪是MTK比較高，之前不知道MTK也有Android，以前以為只有網通
從看到一些強者從Google跳槽到NV後，這張圖好像不會很意外。
這個問我爸後，原來Android不只有手機，許多如smart TV等底層很多都是Android
### 在台灣 Testing 去哪間？
![](https://i.imgur.com/pkM4DZ5.png)
原來Testing是被Apple拉高，大部分頂級外商也都有Testing職位
### 在台灣 ML/AI 去哪間？
![](https://i.imgur.com/AEGhCQZ.png)
在台灣ML給最高是Amazon，意外的是街口竟然比外商及半導體公司高
### 在台灣 Production 去哪間？
![](https://i.imgur.com/YqAGJtM.png)
產品工程師，最多是NV，其次是群暉、MTK，MTK起薪比較高，群暉天花板高一點
### 在台灣 Embeding 去哪間？
![](https://i.imgur.com/qjN1B2s.png)
做嵌入式系統裡，MTK天花板最高，但其薪資差異也最多，其次AWS、Google
### 在台灣 什麼公司開最高？
![](https://i.imgur.com/mWIzls2.png)
頂級外商(FAANG+NV)、MTK開的都很高
### 在台灣 什麼底薪公司開最高？
![](https://i.imgur.com/v4mnaUj.png)
MTK、瑞昱、街口、頂級外商的薪水底薪都很高
### 在台灣 新鮮人什麼公司開最高？
![](https://i.imgur.com/44n95Pq.png)
### 在台灣 新鮮人(YOE &#x3C; 1y)什麼公司底薪開最高？
![](https://i.imgur.com/4EgIPSw.png)
### 在台灣 工作型態對薪資影響
![](https://i.imgur.com/ruPEhPi.png)
### 在台灣 工作年資對薪資影響
![](https://i.imgur.com/pFTecCG.png)
### 在台灣 工作地區對薪資影響
![](https://i.imgur.com/aJovsL7.png)
中位數、p75花蓮最高
### 在台灣 學位對薪資影響
![](https://i.imgur.com/xTOjfth.png)
博士起新高、中位數高，但碩士的極端值最高，p75也最高
## Ref

</body>
</html>


[網頁互動版](https://alanhc.github.io/taiwan_jobs.html)
- 資料來源：Levels Fyi
## 台灣
### 在台灣 軟體工程師 做什麼最賺錢？
![](https://i.imgur.com/GHvjK50.png)
Android先猜應該是因為Google的關係拉高平均，Testing有點意外，Backend也是常常被提到成長幅度較大的，Networking先猜應該是網通廠MTK之類的。
AI 天花板高一點，但整體比做產品、嵌入式系統低。
### 在台灣 軟體工程師 做什麼底薪最好？
![](https://i.imgur.com/TnOjosz.png)
起薪最高Android，但成長幅度最高是Backend，但只有很少一部分

### 在台灣 Android 去哪間？


![](https://i.imgur.com/9hAIRym.png)

果然Google最高，但起薪是MTK比較高，之前不知道MTK也有Android，以前以為只有網通
從看到一些強者從Google跳槽到NV後，這張圖好像不會很意外。
這個問我爸後，原來Android不只有手機，許多如smart TV等底層很多都是Android
### 在台灣 Testing 去哪間？

![](https://i.imgur.com/pkM4DZ5.png)

原來Testing是被Apple拉高，大部分頂級外商也都有Testing職位
### 在台灣 ML/AI 去哪間？

![](https://i.imgur.com/AEGhCQZ.png)

在台灣ML給最高是Amazon，意外的是街口竟然比外商及半導體公司高
### 在台灣 Production 去哪間？

![](https://i.imgur.com/YqAGJtM.png)

產品工程師，最多是NV，其次是群暉、MTK，MTK起薪比較高，群暉天花板高一點
### 在台灣 Embeding 去哪間？
![](https://i.imgur.com/qjN1B2s.png)
做嵌入式系統裡，MTK天花板最高，但其薪資差異也最多，其次AWS、Google
### 在台灣 什麼公司開最高？
![](https://i.imgur.com/mWIzls2.png)
頂級外商(FAANG+NV)、MTK開的都很高

### 在台灣 什麼底薪公司開最高？
![](https://i.imgur.com/v4mnaUj.png)
MTK、瑞昱、街口、頂級外商的薪水底薪都很高 
### 在台灣 新鮮人什麼公司開最高？
![](https://i.imgur.com/44n95Pq.png)
### 在台灣 新鮮人(YOE < 1y)什麼公司底薪開最高？
![](https://i.imgur.com/4EgIPSw.png)
### 在台灣 工作型態對薪資影響
![](https://i.imgur.com/ruPEhPi.png)
### 在台灣 工作年資對薪資影響
![](https://i.imgur.com/pFTecCG.png)
### 在台灣 工作地區對薪資影響
![](https://i.imgur.com/aJovsL7.png)
中位數、p75花蓮最高
### 在台灣 學位對薪資影響
![](https://i.imgur.com/xTOjfth.png)
博士起新高、中位數高，但碩士的極端值最高，p75也最高
## Ref


20240211-taiwan-job-insights

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
設計像是『Pastebin』，一個可以讓使用者輸入文字並且取得分享鏈結的系統
## Solutions
### Step 1: Scope the Problem
- 系統不需要有帳戶功能及編輯檔案
- 系統需要追蹤頁面存取次數
- 過舊的檔案要定時刪除
- 使用者不應該很容易猜到網址
- 除了frontend 還有api
- 統計資料會顯示在stats分頁
### Step 2: Make Reasonable Assumptions
- 系統的traffic十分繁重且包含數以百萬的document
- traffic不是平均分散在不同document
### Step 3: Draw the Major Components
- 如何儲存doc?
- db
- file: 因為檔案可能很大，搜尋要比較快的話存檔案比較好
- 簡單設計方式：
```mermaid
graph TD
A(URL to File Database) --> B(server with files)
A(URL to File Database) --> C(server with files)
A(URL to File Database) --> D(server with files)
```
- db 用來存file location
- db存traffic anlysis(timestamp, IP, address and location)
### Step 4: Identify the Key Issue
- 有部分資料會被頻繁讀取，若存在檔案系統，存取會較花時間，因此可能需要cache，由於不會有更新doc，所以cache不會有資料不合法問題
- 考慮從URL建立hash map的 database sharding
- 問題：如果要新增server，如何redistribute doc?
#### Generating URLs
- 鏈結要比較難被猜到
- 使用GUID，缺點：太長
- 10碼英數混合，容易有碰撞問題
- 可以搭配資料庫查詢是否碰撞
#### Analytics
有兩個選擇
- 當每次瀏覽時存取他的rawdata
- Pros: 資料彈性
- 例子：使用log file
- 只存瀏覽次數
- 如果log資料太大？
- 用機率方式儲存log
- 熱門的url 儲存積蓄小，熱門的url可能10次才存一次log
- 可以按照月份存取總瀏覽數
### Follow-up Questions
- 如何支援帳戶
- 如何新增一個分析的property? e.g. 推薦來源
- 如果stats要顯示在每個doc，會如何改變設計？

</body>
</html>


## Question

設計像是『Pastebin』，一個可以讓使用者輸入文字並且取得分享鏈結的系統
## Solutions
### Step 1: Scope the Problem
- 系統不需要有帳戶功能及編輯檔案
- 系統需要追蹤頁面存取次數
- 過舊的檔案要定時刪除
- 使用者不應該很容易猜到網址
- 除了frontend 還有api
- 統計資料會顯示在stats分頁
### Step 2: Make Reasonable Assumptions
- 系統的traffic十分繁重且包含數以百萬的document
- traffic不是平均分散在不同document
### Step 3: Draw the Major Components
- 如何儲存doc? 
	- db
	- file: 因為檔案可能很大，搜尋要比較快的話存檔案比較好
- 簡單設計方式：

```mermaid
graph TD
A(URL to File Database) -->  B(server with files)
A(URL to File Database) -->  C(server with files)
A(URL to File Database) -->  D(server with files)
```
- db 用來存file location
- db存traffic anlysis(timestamp, IP, address and location)
### Step 4: Identify the Key Issue
- 有部分資料會被頻繁讀取，若存在檔案系統，存取會較花時間，因此可能需要cache，由於不會有更新doc，所以cache不會有資料不合法問題
- 考慮從URL建立hash map的 database sharding
	- 問題：如果要新增server，如何redistribute doc?
#### Generating URLs
- 鏈結要比較難被猜到
	- 使用GUID，缺點：太長
	- 10碼英數混合，容易有碰撞問題
		- 可以搭配資料庫查詢是否碰撞
#### Analytics
有兩個選擇
- 當每次瀏覽時存取他的rawdata
	- Pros: 資料彈性
	- 例子：使用log file
- 只存瀏覽次數
- 如果log資料太大？
	- 用機率方式儲存log
		- 熱門的url 儲存積蓄小，熱門的url可能10次才存一次log
	- 可以按照月份存取總瀏覽數
### Follow-up Questions
- 如何支援帳戶
- 如何新增一個分析的property? e.g. 推薦來源
- 如果stats要顯示在每個doc，會如何改變設計？

20240211-pastebin-design

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
如何打造一個個人金融管理系統（mint.com），可以連結銀行帳戶、分析消費習慣及推薦
## Solutions
### Step 1: Scope the Problem
跟面試官討論
- 可以建立account、可以新增多個銀行帳戶、也可以之後再加
- 可以存取金融歷史，也可以根據銀行
- 金融歷史包括收入、支出、現金、投資
- 金融交易包含類別（食物、旅遊、衣服）
- 使用者可以新增自訂類別
- 使用者會取得根據其在的消費群裡的消費推薦
- 目前只需要網站，app之後討論
- 可能需要根據特定條件使用email 通知使用者
- 使用者不能根據自訂rule對交易進行分類
- 假設類別根據銷售商店，而不是價格或日期
### Step 2: Make Reasonable Assumptions
- 新增、移除帳戶相對少用
- 系統是write-heavy，一班使用者每天都會使用，少部分一週使用一次，通常透過email進入網站
- 交易可以被使用者更改類別，但只針對單筆，而不是所有交易歷史裡同一個交易改變
- 銀行不會自動更新資料到我們系統，我們需要自行從銀行取得
- 超出預算的通知不用立即通知，可以在一天內通知即可
### Step 3: Draw the Major Components
```mermaid
graph TD
A(Bank data synchronizer) --> B(raw transcation data)
B(raw transcation data) --> C(Categorizer)
C(Categorizer) --> D(categorized transcations)
D(categorized transcations) &#x3C;--> E(Frontend)
D(categorized transcations) --> F(Budget analyer)
F(Budget analyer) --> G(Budget data)
E(Frontend) &#x3C;--> G(Budget data)
```
- 帳戶更新取決使用者使用頻率，可能每小時/每天
- 資料一開始先暫存，然後交給Categorizer分類後才存到資料庫
- Budget analyer query 分類交易，並儲存使用者的消費
- 前端存取categorized transcations及Budget data的資料，使用者可以改變Budget、類別進行不同的查詢
### Step 4: Identity the Key Issues
- 這是一個data-heavy的系統，為了讓他可以即時反應，會使用非同步（asynchronous）處理
- 有一個task queue，可能包含：擷取銀行資料、重複分析預算、分類銀行資料
- 將task 分優先順序，並確保低優先順序會執行得到
- 假設有很多人註冊不使用，那可能需要凍結或移除
- 可以非同步、使用不同server執行
#### Categorizer and Budge Analyzer
- 一瞬間可能有很多資料，可能不適合使用中心資料庫（沒效率）
- 使用檔案存取可能優於資料庫，並使用商家的名稱命名
- categorizer 流程可能如下：
```mermaid
graph TD
A(raw transcation data, group by seller) --> B(categorized data, group by user)
B(categorized data, group by user) --> C(merge &#x26; group by user &#x26; category)
B(categorized data, group by user) --> E(update categorized transcations)
C(merge &#x26; group by user &#x26; category) --> D(update budges)
```
- 根據商家分類資料，有可能存在cache
- 再根據使用者group，這些交易再根據使用者新增進資料庫
- 原本：商家>使用者、price、日期
- 處理完：使用者>商家>類別>price、日期
#### User Changing Categories
- 重新計算類別，並且找尋該使用者之前的記錄修改
### Follow up
- 如果要讓他可以支援mobile app，要怎麼設計？
- 如何設計一個component將item 分類
- 如何設計推薦預算的feature
- 使用者若要可以自訂rule分類交易商家，要怎麼做？

</body>
</html>


## Question

如何打造一個個人金融管理系統（mint.com），可以連結銀行帳戶、分析消費習慣及推薦
## Solutions
### Step 1: Scope the Problem
跟面試官討論
- 可以建立account、可以新增多個銀行帳戶、也可以之後再加
- 可以存取金融歷史，也可以根據銀行
- 金融歷史包括收入、支出、現金、投資
- 金融交易包含類別（食物、旅遊、衣服）
- 使用者可以新增自訂類別
- 使用者會取得根據其在的消費群裡的消費推薦
- 目前只需要網站，app之後討論
- 可能需要根據特定條件使用email 通知使用者
- 使用者不能根據自訂rule對交易進行分類
- 假設類別根據銷售商店，而不是價格或日期
### Step 2: Make Reasonable Assumptions
- 新增、移除帳戶相對少用
- 系統是write-heavy，一班使用者每天都會使用，少部分一週使用一次，通常透過email進入網站
- 交易可以被使用者更改類別，但只針對單筆，而不是所有交易歷史裡同一個交易改變
- 銀行不會自動更新資料到我們系統，我們需要自行從銀行取得
- 超出預算的通知不用立即通知，可以在一天內通知即可
### Step 3: Draw the Major Components

```mermaid
graph TD
A(Bank data synchronizer) -->  B(raw transcation data)
B(raw transcation data) --> C(Categorizer)
C(Categorizer) --> D(categorized transcations)
D(categorized transcations) <--> E(Frontend)
D(categorized transcations) --> F(Budget analyer)
F(Budget analyer) --> G(Budget data) 
E(Frontend) <--> G(Budget data) 
```
- 帳戶更新取決使用者使用頻率，可能每小時/每天
- 資料一開始先暫存，然後交給Categorizer分類後才存到資料庫
- Budget analyer query 分類交易，並儲存使用者的消費
- 前端存取categorized transcations及Budget data的資料，使用者可以改變Budget、類別進行不同的查詢
### Step 4: Identity the Key Issues
- 這是一個data-heavy的系統，為了讓他可以即時反應，會使用非同步（asynchronous）處理
- 有一個task queue，可能包含：擷取銀行資料、重複分析預算、分類銀行資料
- 將task 分優先順序，並確保低優先順序會執行得到
- 假設有很多人註冊不使用，那可能需要凍結或移除
- 可以非同步、使用不同server執行
#### Categorizer and Budge Analyzer
- 一瞬間可能有很多資料，可能不適合使用中心資料庫（沒效率）
- 使用檔案存取可能優於資料庫，並使用商家的名稱命名
- categorizer 流程可能如下：
```mermaid
graph TD
A(raw transcation data, group by seller) -->  B(categorized data, group by user)
B(categorized data, group by user) --> C(merge & group by user & category)
B(categorized data, group by user) --> E(update categorized transcations)
C(merge & group by user & category) --> D(update budges)
```
- 根據商家分類資料，有可能存在cache
- 再根據使用者group，這些交易再根據使用者新增進資料庫
- 原本：商家>使用者、price、日期
- 處理完：使用者>商家>類別>price、日期
#### User Changing Categories
- 重新計算類別，並且找尋該使用者之前的記錄修改
### Follow up
- 如果要讓他可以支援mobile app，要怎麼設計？
- 如何設計一個component將item 分類
- 如何設計推薦預算的feature
- 使用者若要可以自訂rule分類交易商家，要怎麼做？

20240207-personal-finanicial-manager

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
設計一個有多類別的產品ranking網站
## Solutions
### Step 1 Scope the problem
- 只跟問題相關、而不是整個系統，包含前端及購買component
- 定義什麼是salesrank，包含前天、上週、上個月
- 假設有多個類別，且不會有子類別
### Step 2 Make Reasonable Assumptions
- 假設不用更新全部資料
- 熱門商品數字要精確、不熱門的可以不用那麼精確
- 熱門資料要每小時更新，其餘的超過7天不用很精確
### Step 3 Draw the Major Component
```mermaid
graph LR
A(Purchase System) --> |orders added to db| B(Database)
B(Database) --> |sort| C(Salesrank Data)
C(Salesrank Data) --> D(Frontend)
A(Purchase System) --> B(Database)
C(Salesrank Data) --> D(Frontend)
```
### Step 4 Identity the Key Issues
- 分析非常昂貴
- 若把資料全部存在一個表，每天會區要更新，所以可以拆分成ID+日期
- example
| Prod ID | Total | Sun | Mon |
| ---- | ---- | ---- | ---- |
| | | | |
| | | | |
| Prod ID | Cat ID |
| --- | --- |
| | |
- 資料庫被頻繁寫入
- 可能會需要batch write，在之前可能先cache在某種memory，但要考慮是否能放進hash table，假設有1000萬筆資料...
- 注意更新sales rank與資料寫入的時間，確保不會有偏差
- join 查詢非常貴
- 與其先將資料裡每個類別sort過再join，會花很多時間，不如一開始使用資料格式
| Prod ID | Cat | Total | Sun | Mon |
| ---- | ---- | ---- | ---- | ---- |
| | | | | |
| | | | | |
| | | | | |
- 資料庫查詢很昂貴
- 如果write/search很昂貴可以使用類似log file像是map reduce
- 每個query使用檔案及資料夾分類
#### Follower up
- 你如何會碰到接下來的bottleneck，如何解釋？
- 如果有子類別？
- 如果資料要很準確，達到30min內都是精確的

</body>
</html>


## Question
設計一個有多類別的產品ranking網站
## Solutions
### Step 1 Scope the problem
- 只跟問題相關、而不是整個系統，包含前端及購買component
- 定義什麼是salesrank，包含前天、上週、上個月
- 假設有多個類別，且不會有子類別
### Step 2 Make Reasonable Assumptions
- 假設不用更新全部資料
- 熱門商品數字要精確、不熱門的可以不用那麼精確
- 熱門資料要每小時更新，其餘的超過7天不用很精確
### Step 3 Draw the Major Component
```mermaid
graph LR
A(Purchase System) --> |orders added to db| B(Database)
B(Database) --> |sort| C(Salesrank Data)
C(Salesrank Data) --> D(Frontend)
A(Purchase System) --> B(Database)
C(Salesrank Data) --> D(Frontend) 
```
### Step 4 Identity the Key Issues
- 分析非常昂貴
	- 若把資料全部存在一個表，每天會區要更新，所以可以拆分成ID+日期
	- example

| Prod ID | Total | Sun | Mon |
| ---- | ---- | ---- | ---- |
|  |  |  |  |
|  |  |  |  |

| Prod ID | Cat ID |
| --- | --- |
|  |  |
- 資料庫被頻繁寫入
	- 可能會需要batch write，在之前可能先cache在某種memory，但要考慮是否能放進hash table，假設有1000萬筆資料...
	- 注意更新sales rank與資料寫入的時間，確保不會有偏差
- join 查詢非常貴
	- 與其先將資料裡每個類別sort過再join，會花很多時間，不如一開始使用資料格式

| Prod ID | Cat | Total | Sun | Mon |
| ---- | ---- | ---- | ---- | ---- |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
- 資料庫查詢很昂貴
	- 如果write/search很昂貴可以使用類似log file像是map reduce
		- 每個query使用檔案及資料夾分類
#### Follower up
- 你如何會碰到接下來的bottleneck，如何解釋？
- 如果有子類別？
- 如果資料要很準確，達到30min內都是精確的


20240205-salesrank-design

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

[Python自動化交易](https://youtu.be/eDolz-_Ls4o?si=SWWnmwDFPey1cUHS)
https://www.xq.com.tw/XQlite-Download.aspx
https://ithelp.ithome.com.tw/users/20129306/ironman/3071?page=2
## Ref

</body>
</html>


[Python自動化交易](https://youtu.be/eDolz-_Ls4o?si=SWWnmwDFPey1cUHS)
https://www.xq.com.tw/XQlite-Download.aspx
https://ithelp.ithome.com.tw/users/20129306/ironman/3071?page=2
## Ref


20240203-trading-bot

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

- [Elizabeth Sloane](https://www.imdb.com/name/nm1567113/?ref_=tt_ch) : Lobbying is about foresight. About anticipating your opponent's moves and devising counter measures. The winner plots one step ahead of the opposition. And plays her trump card just after they play theirs. It's about making sure you surprise them. And they don't surprise you.
- - [Cynthia](https://www.imdb.com/name/nm2543501/?ref_=tt_ch) : That is so cynical
[Elizabeth Sloane](https://www.imdb.com/name/nm1567113/?ref_=tt_ch) : Cynical is a word used by Pollyannas to denote an absence of the naiveté they so keenly exhibit.
## Ref

</body>
</html>


- [Elizabeth Sloane](https://www.imdb.com/name/nm1567113/?ref_=tt_ch) : Lobbying is about foresight. About anticipating your opponent's moves and devising counter measures. The winner plots one step ahead of the opposition. And plays her trump card just after they play theirs. It's about making sure you surprise them. And they don't surprise you.
- - [Cynthia](https://www.imdb.com/name/nm2543501/?ref_=tt_ch) : That is so cynical
    [Elizabeth Sloane](https://www.imdb.com/name/nm1567113/?ref_=tt_ch) : Cynical is a word used by Pollyannas to denote an absence of the naiveté they so keenly exhibit.
    
## Ref


quotes

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
再設計爬蟲時，如何避免無限迴圈
## Solutions
問題：無限迴圈何時發生？因為我們將鏈結化作graph，所以有可能有無限迴圈，有可能使用`hashtable[v]=true`及BFS ，但v怎麼設定？
但遇到URL parameters可能會有問題，比如example.com?pid=google及example.com?pid=apple可能是不同，但example.com?foo=hi 跟example.com是相同網頁
根據內容？使用 相似度，首先爬取children，計算子頁相似度極優先度，如下
1. 開啟網頁、建立page signature根據 頁面、url
2. query database看簽名是否最近被爬取過
3. 如果有，新增到db到low priority
4. 如果沒有，爬取網頁並新增資料庫

</body>
</html>


## Question
再設計爬蟲時，如何避免無限迴圈


## Solutions
問題：無限迴圈何時發生？因為我們將鏈結化作graph，所以有可能有無限迴圈，有可能使用`hashtable[v]=true`及BFS ，但v怎麼設定？
但遇到URL parameters可能會有問題，比如example.com?pid=google及example.com?pid=apple可能是不同，但example.com?foo=hi 跟example.com是相同網頁
根據內容？使用 相似度，首先爬取children，計算子頁相似度極優先度，如下
1. 開啟網頁、建立page signature根據 頁面、url
2. query database看簽名是否最近被爬取過
3. 如果有，新增到db到low priority
4. 如果沒有，爬取網頁並新增資料庫


20240125-web_srawler_design

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
如何儲存1000萬的url，每個字4 byte、頁面平均100字，這樣大約會有4TB
## Solutions
### Solution 1: Disk Storage
使用two pass
first pass: 將URL hash 儲存，分成 每個1G，共4000chunk，儲存URL u到`
<div>
  .txt`
  `
  <div>
    .txt where x = hash(u%4000)`
    second pass: 讀入memory、建立hash table、尋找重複
    ### Solution 2: Multiple Machines
    跟two pass一樣，只是在不同機器
    - Pros
    - 可以平行處理，對大型資料效率佳
    - Cons
    - 複雜、維護多個機器困難
  </div>
</div>

</body>
</html>


## Question
如何儲存1000萬的url，每個字4 byte、頁面平均100字，這樣大約會有4TB


## Solutions

### Solution 1: Disk Storage
使用two pass
first pass: 將URL hash 儲存，分成 每個1G，共4000chunk，儲存URL u到`<x>.txt`
`<x>.txt where x = hash(u%4000)`
second pass: 讀入memory、建立hash table、尋找重複
### Solution 2: Multiple Machines
跟two pass一樣，只是在不同機器
- Pros
	- 可以平行處理，對大型資料效率佳
- Cons
	- 複雜、維護多個機器困難


20240125-duplicate-url-design

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
設計一個簡單搜尋引擎的webserver，假設系統有100台機器負責回應queries，有個昂貴的function，processSearch(string query)，機器的選擇是隨機的，同一台機器回應可能會不一樣，請設計最相關queries的caching機制，請解釋當資料改變時如何更新cache
## Solutions
與面試官討論假設
### Assumptions
沒有最正確的答案
- 所有的query 發生在機器最初呼叫時
- cache的queries要有百萬等級
- 在機器間呼叫要相對快
- 結果是order list，並且有50字以內title及200字summary
- 最熱門的會極端熱門，而且隨時都應該在cache
### System Requirements
- 根據key有效檢索
- 過期舊資料應該被刪除
### Step 1 design a cache for a single system
如何有效根據key新增及刪除資料？
- lined list 可以有效快速刪除特定節點、限制長度
- hashtable可以有效找尋data
使用hashmap map query 到特定node
### Step 2 Expand to many machines
#### Option 1: Each machine has its own cache
- Pros
- 相對快、因為沒有machine-to-machine call
- Cons
- 如果有很多重複的query會很沒有效率
#### Option 2: Each machine has a copy of the cache
- Pros
- 在cache一定會找得到
- Cons
- 更新、佔用空間大、cache總數會較小
#### Option 3: Each machine stores a segment of the cache
根據 `hash(query)%N`
- Pros
- 相對可儲存多
- Cons
- 增加machine-to-machine call
### Step 3 Updating results when contents change
數量大、太熱門會需要時間快取或按照需求快取(和interviewer討論)
確認什麼時候資料會怎麼改變
1. URL 裡面內容 改變
2. 根據page ranking改變
3. 根據特定query改變
1,2 可以使用hash table，根據特定URL cache query
資料不需同步的話，可以根據時間爬取
3 可以再快取裡面使用 auto-matic time out，只cache特定時間，否則timeout，可以確保資料定時更新
### Step 4: Further Enhancements
轉送情形可以在中間節點cache
重新設計架構，不是隨機導流，而是使用hash
導入資料time out機制，刪除過久資料

</body>
</html>


## Question
設計一個簡單搜尋引擎的webserver，假設系統有100台機器負責回應queries，有個昂貴的function，processSearch(string query)，機器的選擇是隨機的，同一台機器回應可能會不一樣，請設計最相關queries的caching機制，請解釋當資料改變時如何更新cache

## Solutions
與面試官討論假設
### Assumptions
沒有最正確的答案
- 所有的query 發生在機器最初呼叫時
- cache的queries要有百萬等級
- 在機器間呼叫要相對快
- 結果是order list，並且有50字以內title及200字summary
- 最熱門的會極端熱門，而且隨時都應該在cache
### System Requirements
- 根據key有效檢索
- 過期舊資料應該被刪除
### Step 1 design a cache for a single system
如何有效根據key新增及刪除資料？
- lined list 可以有效快速刪除特定節點、限制長度
- hashtable可以有效找尋data
使用hashmap map query 到特定node
### Step 2 Expand to many machines
#### Option 1: Each machine has its own cache
- Pros
	- 相對快、因為沒有machine-to-machine call
- Cons
	- 如果有很多重複的query會很沒有效率
#### Option 2: Each machine has a copy of the cache
- Pros
	- 在cache一定會找得到
- Cons
	- 更新、佔用空間大、cache總數會較小
#### Option 3: Each machine stores a segment of the cache
根據 `hash(query)%N`
- Pros
	- 相對可儲存多
- Cons
	- 增加machine-to-machine call
### Step 3 Updating results when contents change
數量大、太熱門會需要時間快取或按照需求快取(和interviewer討論)
確認什麼時候資料會怎麼改變
1. URL 裡面內容 改變
2. 根據page ranking改變
3. 根據特定query改變
1,2 可以使用hash table，根據特定URL cache query
資料不需同步的話，可以根據時間爬取
3 可以再快取裡面使用 auto-matic time out，只cache特定時間，否則timeout，可以確保資料定時更新
### Step 4: Further Enhancements
轉送情形可以在中間節點cache
重新設計架構，不是隨機導流，而是使用hash
導入資料time out機制，刪除過久資料


20240125-cache-design

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

今日看到好玩的東西，改天有時間來玩
## Ref
- https://github.com/g1879/DrissionPage

</body>
</html>


今日看到好玩的東西，改天有時間來玩
## Ref
- https://github.com/g1879/DrissionPage

20240124-break-cloudflare-bot-prevention

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

論文研究如何避免黃牛的賣票策略，其實也順便研究了黃牛
## 黃牛訪談影片
[【紀錄片】黃牛末日？直擊台灣黃牛產業鏈 - 第九人稱](https://youtu.be/m2gGWCkKkIM?si=0T8543LItJcNVrY7)
[【國昌直播】看看上週立法院在做什麼_0410 ｜連線前黃牛](https://www.youtube.com/live/cskG3nfWXjs?si=Tth1Bfr-loNKDAGM)
## bot
https://github.com/max32002/tixcraft_bot?tab=readme-ov-file
## Ref

</body>
</html>


論文研究如何避免黃牛的賣票策略，其實也順便研究了黃牛

## 黃牛訪談影片
[【紀錄片】黃牛末日？直擊台灣黃牛產業鏈 - 第九人稱](https://youtu.be/m2gGWCkKkIM?si=0T8543LItJcNVrY7)
[【國昌直播】看看上週立法院在做什麼_0410 ｜連線前黃牛](https://www.youtube.com/live/cskG3nfWXjs?si=Tth1Bfr-loNKDAGM)
## bot
https://github.com/max32002/tixcraft_bot?tab=readme-ov-file
## Ref


20240123-scalper

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Question
設計一個可以有1000 client 可以取得最後的開高收低，已經有資料、可以自行決定儲存，要怎麼設計client說明幾種不同的方法及為甚麼使用，可以用任意技術及可以選擇任意分散機制
## Solutions
先考慮以下幾點
- Client Ease of use: 對顧客使用簡單且有用
- Ease for Our selves: 對我們來說好實作且好維護
- Flexibility for future demand: 如何根據真實世界需求彈性調整
- Scalability and Efficiency: 注意解決有效性
### Propsal 1 file
使用簡單text file，且讓顧客透過FTP下載
- Pros
- 簡單好維護、方便閱讀、備份
- Cons
- 搜尋、排序困難
- 新增資料會break paring mechanism
### Propsal 2 SQL
- Pros
- 方便查詢且有效率
- rolling back, backing up, security 已經有，不用重新造輪子
- 方便整合現有應用，因為SQL幾乎是軟體開發標準
- Cons
- 笨重，會複雜化系統
- 人類難閱讀
- 對於client要注意、不該給的權限、使否執行昂貴、沒效率的queries
### Propsal 3 XML
- Pros
- easy to distirbute, 對機器、人類好閱讀
- 大部分程式都支援XML
- 很好新增 插入
- 有很多工具可以back up
- Cons
- 執行query 需要取得全部data
- client 會有全部資料，會沒有效率
沒有正確的答案，雖然看起來file的做法最不好，每種方法都有trade off
## Ref

</body>
</html>


## Question
設計一個可以有1000 client 可以取得最後的開高收低，已經有資料、可以自行決定儲存，要怎麼設計client說明幾種不同的方法及為甚麼使用，可以用任意技術及可以選擇任意分散機制
## Solutions
先考慮以下幾點
- Client Ease of use: 對顧客使用簡單且有用
- Ease for Our selves: 對我們來說好實作且好維護
- Flexibility for future demand: 如何根據真實世界需求彈性調整
- Scalability and Efficiency: 注意解決有效性 
### Propsal 1 file
使用簡單text file，且讓顧客透過FTP下載
- Pros
	- 簡單好維護、方便閱讀、備份
- Cons
	- 搜尋、排序困難
	- 新增資料會break paring mechanism
### Propsal 2 SQL
- Pros
	- 方便查詢且有效率
	- rolling back, backing up, security 已經有，不用重新造輪子
	- 方便整合現有應用，因為SQL幾乎是軟體開發標準
- Cons
	- 笨重，會複雜化系統
	- 人類難閱讀
	- 對於client要注意、不該給的權限、使否執行昂貴、沒效率的queries
### Propsal 3 XML
- Pros
	- easy to distirbute, 對機器、人類好閱讀
	- 大部分程式都支援XML
	- 很好新增 插入
	- 有很多工具可以back up 
- Cons
	- 執行query 需要取得全部data
	- client 會有全部資料，會沒有效率
沒有正確的答案，雖然看起來file的做法最不好，每種方法都有trade off


## Ref


20240122-stock-data

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Ref
- https://www.youtube.com/watch?v=hT6A8zRn0Ck&#x26;t=31s&#x26;ab_channel=%E9%82%8A%E7%B7%A3%E4%BA%BA%E7%9A%84%E5%9E%83%E5%9C%BE%E6%A1%B6

</body>
</html>



## Ref
- https://www.youtube.com/watch?v=hT6A8zRn0Ck&t=31s&ab_channel=%E9%82%8A%E7%B7%A3%E4%BA%BA%E7%9A%84%E5%9E%83%E5%9C%BE%E6%A1%B6


20240122-ntu-cool-video-download

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

[Voice Control for ChatGPT](https://chromewebstore.google.com/detail/voice-control-for-chatgpt/eollffkcakegifhacjnlnegohfdlidhn)
## Ref
- https://youtu.be/UPh0iQbg4iU?si=WVHhbe66odXGZaUK

</body>
</html>


[Voice Control for ChatGPT](https://chromewebstore.google.com/detail/voice-control-for-chatgpt/eollffkcakegifhacjnlnegohfdlidhn)

## Ref
- https://youtu.be/UPh0iQbg4iU?si=WVHhbe66odXGZaUK

沒環境講英文？如何跟 ChatGPT 對話，快速加強口說

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

- https://github.com/alanhc/bot/blob/main/instagram.ipynb
## Ref
https://youtu.be/iJGvYBH9mcY?si=lVPMP3EG68ppn5y_

</body>
</html>


- https://github.com/alanhc/bot/blob/main/instagram.ipynb
## Ref
https://youtu.be/iJGvYBH9mcY?si=lVPMP3EG68ppn5y_

20240121-Web-Scraping-Instagram-with-Selenium

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

- 一開始看到
- [The KK Show - 187 中共網軍的敵人 - 沈伯洋](https://youtu.be/Da2LDllZjHQ?si=KfYkfcB_ok7PZ_VH)
- [【賀瓏夜夜秀】沈伯洋 被攻擊炎上是榮幸](https://youtu.be/eiBJmdjTOwU?si=HhDiyqeQFqkg7MT_)
- [The KK Show - 34 小粉紅殺手 - 沈伯洋（助理）教授](https://youtu.be/XCBjCtCNcSQ?si=MRj2qceUZOXN-jqd)
- 攻敵必救、遊說、關鍵人物、
- [網軍常見手法全面解析，製造網路恐懼效應，應立法管制嗎？【時事評判】](https://youtu.be/NSt4RUKkkik?si=U-urp9djmwXdl-KR)
- 相關文章
- [【滲透測試LAB】如何使用OSINT - SpiderFoot 執行情蒐](https://minmin0625.medium.com/滲透測試lab-如何使用osint-spiderfoot-執行情蒐-425de6e9198c)
- [肉搜神器theHarvester　學APT攻擊找開源情資](https://www.netadmin.com.tw/netadmin/zh-tw/technology/E6BE51392849415BA8F586310A091CA5)
https://www.nss.com.tw/socialmarketing/product/ptt%E9%96%8B%E6%96%87%E4%B8%80%E9%96%8B%E4%BA%94%E5%9B%9E%E6%AF%8F%E7%AF%87/
https://sectools.tw/sherlock/
[簡單操作 Facebook 爬蟲：Facebook scraper 套件介紹](https://vocus.cc/article/63cd303ffd89780001d86678)
https://github.com/kevinzg/facebook-scraper
https://github.com/topics/osint
https://github.com/topics/osint-tools
https://github.com/jivoi/awesome-osint
https://github.com/alishahlakhani/Facebook-comments-exporter-chrome-extension/blob/master/src/scripts/background.js: 擷取facebook comments
https://www.youtube.com/watch?v=2MJt1dTqbPk&#x26;ab_channel=%E5%B0%8FLin%E8%AF%B4: 擷取facebook comments
## sherlock (從帳號名稱找註冊網站)
- 可以使用
https://github.com/sherlock-project/sherlock
## Twint （從twitter找貼文）
- 不確定是否可以使用
Analyzing Tweets with NLP in minutes with Spark, Optimus and Twint
[How to use Twint as an OSINT tool](https://pielco11.ovh/posts/twint-osint/)
## Osintagram (Instagram osint工具)
- 可以使用
https://github.com/Datalux/Osintgram?tab=readme-ov-file
## Ref

</body>
</html>


- 一開始看到
	- [The KK Show - 187 中共網軍的敵人 - 沈伯洋](https://youtu.be/Da2LDllZjHQ?si=KfYkfcB_ok7PZ_VH)
	- [【賀瓏夜夜秀】沈伯洋 被攻擊炎上是榮幸](https://youtu.be/eiBJmdjTOwU?si=HhDiyqeQFqkg7MT_)
	- [The KK Show - 34 小粉紅殺手 - 沈伯洋（助理）教授](https://youtu.be/XCBjCtCNcSQ?si=MRj2qceUZOXN-jqd)
	- 攻敵必救、遊說、關鍵人物、
	- [網軍常見手法全面解析，製造網路恐懼效應，應立法管制嗎？【時事評判】](https://youtu.be/NSt4RUKkkik?si=U-urp9djmwXdl-KR)
- 相關文章
	- [【滲透測試LAB】如何使用OSINT - SpiderFoot 執行情蒐](https://minmin0625.medium.com/滲透測試lab-如何使用osint-spiderfoot-執行情蒐-425de6e9198c)
	- [肉搜神器theHarvester　學APT攻擊找開源情資](https://www.netadmin.com.tw/netadmin/zh-tw/technology/E6BE51392849415BA8F586310A091CA5)
https://www.nss.com.tw/socialmarketing/product/ptt%E9%96%8B%E6%96%87%E4%B8%80%E9%96%8B%E4%BA%94%E5%9B%9E%E6%AF%8F%E7%AF%87/
https://sectools.tw/sherlock/

[簡單操作 Facebook 爬蟲：Facebook scraper 套件介紹](https://vocus.cc/article/63cd303ffd89780001d86678)
https://github.com/kevinzg/facebook-scraper
https://github.com/topics/osint
https://github.com/topics/osint-tools
https://github.com/jivoi/awesome-osint
https://github.com/alishahlakhani/Facebook-comments-exporter-chrome-extension/blob/master/src/scripts/background.js: 擷取facebook comments
https://www.youtube.com/watch?v=2MJt1dTqbPk&ab_channel=%E5%B0%8FLin%E8%AF%B4: 擷取facebook comments
## sherlock (從帳號名稱找註冊網站)
- 可以使用
https://github.com/sherlock-project/sherlock

## Twint （從twitter找貼文）
- 不確定是否可以使用
Analyzing Tweets with NLP in minutes with Spark, Optimus and Twint
[How to use Twint as an OSINT tool](https://pielco11.ovh/posts/twint-osint/)

## Osintagram (Instagram osint工具)
- 可以使用
https://github.com/Datalux/Osintgram?tab=readme-ov-file

## Ref


20240119-OSINT

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

在台灣，跟商品有關的統計資料價格大約一年100w (客製化程度不同)
統計資訊包含 競品分析 競爭對手售價 客製化報告等等
why? 品牌商想知道自己跟對手銷售情形
線上臨售 -(raw)-> OO Corp. -(客製化分析報告)-> 商品品牌
台灣市場規模不大
馬來西亞 數值輸入 -> 台灣整理+分析
加拿大 diploma -> 美國工作
## Ref
- 跟別人聊天

</body>
</html>


在台灣，跟商品有關的統計資料價格大約一年100w (客製化程度不同)
統計資訊包含 競品分析 競爭對手售價 客製化報告等等
why? 品牌商想知道自己跟對手銷售情形
線上臨售 -(raw)-> OO Corp. -(客製化分析報告)-> 商品品牌
台灣市場規模不大
馬來西亞 數值輸入 -> 台灣整理+分析 
加拿大 diploma  -> 美國工作
## Ref
- 跟別人聊天


20240114-product-insight

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

以下內容為該影片的筆記：
中國經濟變差 政府監管 員工信心
全球資產配置 美股 賺美元全權資產配置
大廠 推行要一年
大廠大家做得跟核心業務沒關
數據 有博士會有加成、做得好不需要博士、性價比不高
有用的DS?
第一份工作很重要 拿到最好的工作
儘早感受真實世界在想什麼比較重要
面試
business sense
統計、SQL、python、analyical、business（商業問題化成數據可解決）、product sense（找重點）
1. 有沒有思考足夠多的trade-off
2. 有沒有把東西推到落地
3. 這個方法的理解是否是本質的理解
ＤＳ學徒制學習
統計理論很重要
Scientist：用科學方法論理解世界、data &#x26; model 只是輔助
ML scientist pay 會多一些 很多聰明人擠 apply跟發揮價值很少
懂業務的analytics非常少
## Ref
- https://youtu.be/A8GWS00nYDQ?si=mv4ORUgu4cxO45d1

</body>
</html>


以下內容為該影片的筆記：

中國經濟變差 政府監管 員工信心

全球資產配置 美股 賺美元全權資產配置

大廠 推行要一年

大廠大家做得跟核心業務沒關

數據 有博士會有加成、做得好不需要博士、性價比不高

有用的DS?

第一份工作很重要 拿到最好的工作

儘早感受真實世界在想什麼比較重要

面試

business sense

統計、SQL、python、analyical、business（商業問題化成數據可解決）、product sense（找重點）

1. 有沒有思考足夠多的trade-off
2. 有沒有把東西推到落地
3. 這個方法的理解是否是本質的理解

ＤＳ學徒制學習

統計理論很重要

Scientist：用科學方法論理解世界、data & model 只是輔助

ML scientist pay 會多一些 很多聰明人擠 apply跟發揮價值很少

懂業務的analytics非常少
## Ref
- https://youtu.be/A8GWS00nYDQ?si=mv4ORUgu4cxO45d1


20240113-data-pay

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

公司 → 搶奪注意力
太多分心事
- 克服
- 決心、自我反省
why? 手機成癮
多巴安 → 採取行動獲得獎勵
看手機→獎勵→空虛 （重複）久了會疲乏 →思考力下降 高度專注任務難以進行 因為需要相同刺激
多巴安排毒 = 減少外在刺激
1. （激烈）24-48 小時內 斷絕 網路 社群媒體 手機遊戲 ，使用散步 冥想替代
2. 部分排毒：找出最容易分心刺激來源，有意識限制，減少使用時間
1. 找出最容易成癮的 逛網購、短影音
2. 找出想分析原因 戒掉衝動 想為什麼想滑？吸引？當下事情太困難
3. 解決可以解決當前問題的替換事項 找到有停止點的替換事項 e.g. 書 拉筋 放鬆 保持客觀、不批判 感受情緒
4. 延遲享樂 自我控制
1. 現代 太習慣立即性獎賞
2. 滑手機 太習慣立即性想法 影響人際 思考現在的事情對長遠影響 好的留不好的改
5. 有意識計劃每一天：減少分心
1. 早晨看 計畫 e.g. notion ，而不是看email等 → 都在處理別人事情的陷阱
2. 明確的目標、計畫
3. 把零碎時間填滿
## Ref
- https://youtu.be/gvOOgqcpbvU?si=GVwLMdc0DqtdpWC2

</body>
</html>


公司 → 搶奪注意力

太多分心事

- 克服
    - 決心、自我反省

why? 手機成癮

多巴安 → 採取行動獲得獎勵

看手機→獎勵→空虛 （重複）久了會疲乏 →思考力下降 高度專注任務難以進行 因為需要相同刺激

多巴安排毒 = 減少外在刺激

1. （激烈）24-48 小時內 斷絕 網路 社群媒體 手機遊戲 ，使用散步 冥想替代
2. 部分排毒：找出最容易分心刺激來源，有意識限制，減少使用時間
    1. 找出最容易成癮的 逛網購、短影音
    2. 找出想分析原因 戒掉衝動 想為什麼想滑？吸引？當下事情太困難
    3. 解決可以解決當前問題的替換事項 找到有停止點的替換事項 e.g. 書 拉筋 放鬆 保持客觀、不批判 感受情緒
    4. 延遲享樂 自我控制
        1. 現代 太習慣立即性獎賞
        2. 滑手機 太習慣立即性想法 影響人際 思考現在的事情對長遠影響 好的留不好的改
    5. 有意識計劃每一天：減少分心
        1. 早晨看 計畫 e.g. notion ，而不是看email等 → 都在處理別人事情的陷阱
        2. 明確的目標、計畫
        3. 把零碎時間填滿

## Ref
- https://youtu.be/gvOOgqcpbvU?si=GVwLMdc0DqtdpWC2

20240108-attention

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

使用這套件可以讓notion嵌入到網站
因為不能使用iframe 將notion嵌入，而大多數的solution都要付費
## Ref
- https://github.com/NotionX/react-notion-x
- https://github.com/NotionX/react-notion-x/tree/master/examples/minimal

</body>
</html>


使用這套件可以讓notion嵌入到網站
因為不能使用iframe 將notion嵌入，而大多數的solution都要付費
## Ref
- https://github.com/NotionX/react-notion-x
- https://github.com/NotionX/react-notion-x/tree/master/examples/minimal

20240107-notion-websire

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

https://signal.org/blog/facebook-messenger/
https://github.com/signalapp/libsignal-protocol-javascript
## Ref
https://socket.io/docs/v4/tutorial/ending-notes

</body>
</html>


https://signal.org/blog/facebook-messenger/
https://github.com/signalapp/libsignal-protocol-javascript
## Ref
https://socket.io/docs/v4/tutorial/ending-notes

20231222-socket-io

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## 先請AI生成資料
### prompt
```text
請生成三組男女，各有各的自我介紹，僅說明自己，這個叫做content 請將這三組男女配對 用第一個使用者去敘述他配對的對象，這叫做mate 生成以上敘述的json資料，共會有六筆，有mate, content, gender, wish_gender欄位
```
### 回傳
```json
{
"mate":"我心目中的理想伴侶是一個善良、聰明、有趣的人。她應該是一個有自己興趣愛好，能夠獨立思考和行動的人。她應該也是一個幽默風趣的人，能夠讓我的生活充滿樂趣。",
"content":"我是一名 25 歲的男性軟體工程師。我喜歡閱讀、寫作和旅行。我是一個善良、聰明、有趣的人，希望找到一個和我一樣的人。",
"gender":"male",
"wish_gender":"female"
},
{
"mate":"我心目中的理想伴侶是一個陽光、幽默、有責任心的人。他應該是一個熱愛生活，積極向上的人。他應該也是一個有責任心的人，能夠照顧好我和家庭。",
"content":"我是一名 24 歲的女性護士。我喜歡烹飪、看電影和逛街。我是一個陽光、幽默、有責任心的人，希望找到一個和我一樣的人。",
"gender":"female",
"wish_gender":"male"
},
{
"mate":"我心目中的理想伴侶是一個溫柔、體貼、有愛心的女性。她應該是一個善解人意，能夠包容我的人。她應該也是一個有愛心的女性，能夠與我一起去關愛他人。",
"content":"我是一名 26 歲的男性律師。我喜歡運動、打球和聽音樂。我是一個溫柔、體貼、有愛心的人，希望找到一個和我一樣的人。",
"gender":"male",
"wish_gender":"female"
},
{
"mate":"我心目中的理想伴侶是一個善良、聰明、有趣的人。她應該是一個有自己興趣愛好，能夠獨立思考和行動的人。她應該也是一個幽默風趣的人，能夠讓我的生活充滿樂趣。",
"content":"我是一名 25 歲的女性護士。我喜歡烹飪、看電影和逛街。我是一個善良、聰明、有趣的人，希望找到一個和我一樣的人。",
"gender":"female",
"wish_gender":"male"
},
{
"mate":"我心目中的理想伴侶是一個陽光、幽默、有責任心的人。他應該是一個熱愛生活，積極向上的人。他應該也是一個有責任心的人，能夠照顧好我和家庭。",
"content":"我是一名 26 歲的男性軟體工程師。我喜歡運動、打球和聽音樂。我是一個陽光、幽默、有責任心的人，希望找到一個和我一樣的人。",
"gender":"male",
"wish_gender":"female"
}
```
## Vector Database
```python
%load_ext dotenv
%dotenv
import weaviate, os
# Connect with Weaviate Embedded
client = weaviate.connect_to_embedded(
version="1.22.4",
headers={
"X-OpenAI-Api-Key": os.environ[
"OPENAI_API_KEY"
], # Replace with your inference API key
},
)
client.is_ready()
```
```python
import weaviate.classes as wvc
if client.collections.exists("Dating"):
client.collections.delete("Dating")
client.collections.create(
name="Dating",
vectorizer_config=wvc.Configure.Vectorizer.text2vec_openai(), # wvc.Configure.Vectorizer.text2vec_cohere()
)
```
### Insert
```python
questions = client.collections.get("Dating")
questions.data.insert_many(data)
```
## search
```python
print(data[0]["mate"])
from weaviate.classes import Filter
response = questions.query.hybrid(
data[0]["mate"],
alpha=0.7,
filters=(
Filter("gender").equal(data[0]["wish_gender"])
&#x26; (
Filter("wish_gender").equal(data[0]["gender"])
| Filter("wish_gender").equal("others")
)
),
limit=2,
)
for item in response.objects:
print(item.properties)
```
```text
我心目中的理想伴侶是一個善良、聰明、有趣的人。她應該是一個有自己興趣愛好，能夠獨立思考和行動的人。她應該也是一個幽默風趣的人，能夠讓我的生活充滿樂趣。 {'mate': '我心目中的理想伴侶是一個善良、聰明、有趣的人。她應該是一個有自己興趣愛好，能夠獨立思考和行動的人。她應該也是一個幽默風趣的人，能夠讓我的生活充滿樂趣。', 'wish_gender': 'others', 'gender': 'female', 'content': '我是一名 25 歲的女性護士。我喜歡烹飪、看電影和逛街。我是一個善良、聰明、有趣的人，希望找到一個和我一樣的人。'} {'content': '我是一名 24 歲的女性護士。我喜歡烹飪、看電影和逛街。我是一個陽光、幽默、有責任心的人，希望找到一個和我一樣的人。', 'wish_gender': 'male', 'gender': 'female', 'mate': '我心目中的理想伴侶是一個陽光、幽默、有責任心的人。他應該是一個熱愛生活，積極向上的人。他應該也是一個有責任心的人，能夠照顧好我和家庭。'}
```
## Ref
-

</body>
</html>


## 先請AI生成資料
### prompt
```text
請生成三組男女，各有各的自我介紹，僅說明自己，這個叫做content 請將這三組男女配對 用第一個使用者去敘述他配對的對象，這叫做mate 生成以上敘述的json資料，共會有六筆，有mate, content, gender, wish_gender欄位
```
### 回傳
```json
{
   "mate":"我心目中的理想伴侶是一個善良、聰明、有趣的人。她應該是一個有自己興趣愛好，能夠獨立思考和行動的人。她應該也是一個幽默風趣的人，能夠讓我的生活充滿樂趣。",
   "content":"我是一名 25 歲的男性軟體工程師。我喜歡閱讀、寫作和旅行。我是一個善良、聰明、有趣的人，希望找到一個和我一樣的人。",
   "gender":"male",
   "wish_gender":"female"
},
{
   "mate":"我心目中的理想伴侶是一個陽光、幽默、有責任心的人。他應該是一個熱愛生活，積極向上的人。他應該也是一個有責任心的人，能夠照顧好我和家庭。",
   "content":"我是一名 24 歲的女性護士。我喜歡烹飪、看電影和逛街。我是一個陽光、幽默、有責任心的人，希望找到一個和我一樣的人。",
   "gender":"female",
   "wish_gender":"male"
},
{
   "mate":"我心目中的理想伴侶是一個溫柔、體貼、有愛心的女性。她應該是一個善解人意，能夠包容我的人。她應該也是一個有愛心的女性，能夠與我一起去關愛他人。",
   "content":"我是一名 26 歲的男性律師。我喜歡運動、打球和聽音樂。我是一個溫柔、體貼、有愛心的人，希望找到一個和我一樣的人。",
   "gender":"male",
   "wish_gender":"female"
},
{
   "mate":"我心目中的理想伴侶是一個善良、聰明、有趣的人。她應該是一個有自己興趣愛好，能夠獨立思考和行動的人。她應該也是一個幽默風趣的人，能夠讓我的生活充滿樂趣。",
   "content":"我是一名 25 歲的女性護士。我喜歡烹飪、看電影和逛街。我是一個善良、聰明、有趣的人，希望找到一個和我一樣的人。",
   "gender":"female",
   "wish_gender":"male"
},
{
   "mate":"我心目中的理想伴侶是一個陽光、幽默、有責任心的人。他應該是一個熱愛生活，積極向上的人。他應該也是一個有責任心的人，能夠照顧好我和家庭。",
   "content":"我是一名 26 歲的男性軟體工程師。我喜歡運動、打球和聽音樂。我是一個陽光、幽默、有責任心的人，希望找到一個和我一樣的人。",
   "gender":"male",
   "wish_gender":"female"
}
```
## Vector Database
```python
%load_ext dotenv
%dotenv
import weaviate, os

# Connect with Weaviate Embedded
client = weaviate.connect_to_embedded(
    version="1.22.4",
    headers={
        "X-OpenAI-Api-Key": os.environ[
            "OPENAI_API_KEY"
        ],  # Replace with your inference API key
    },
)

client.is_ready()
```

```python
import weaviate.classes as wvc

if client.collections.exists("Dating"):
    client.collections.delete("Dating")
client.collections.create(
    name="Dating",
    vectorizer_config=wvc.Configure.Vectorizer.text2vec_openai(),  # wvc.Configure.Vectorizer.text2vec_cohere()
)

```
### Insert
```python
questions = client.collections.get("Dating")
questions.data.insert_many(data)
```
## search
```python
print(data[0]["mate"])
from weaviate.classes import Filter

response = questions.query.hybrid(
    data[0]["mate"],
    alpha=0.7,
    filters=(
        Filter("gender").equal(data[0]["wish_gender"])
        & (
            Filter("wish_gender").equal(data[0]["gender"])
            | Filter("wish_gender").equal("others")
        )
    ),
    limit=2,
)

for item in response.objects:
    print(item.properties)
```

```text
我心目中的理想伴侶是一個善良、聰明、有趣的人。她應該是一個有自己興趣愛好，能夠獨立思考和行動的人。她應該也是一個幽默風趣的人，能夠讓我的生活充滿樂趣。 {'mate': '我心目中的理想伴侶是一個善良、聰明、有趣的人。她應該是一個有自己興趣愛好，能夠獨立思考和行動的人。她應該也是一個幽默風趣的人，能夠讓我的生活充滿樂趣。', 'wish_gender': 'others', 'gender': 'female', 'content': '我是一名 25 歲的女性護士。我喜歡烹飪、看電影和逛街。我是一個善良、聰明、有趣的人，希望找到一個和我一樣的人。'} {'content': '我是一名 24 歲的女性護士。我喜歡烹飪、看電影和逛街。我是一個陽光、幽默、有責任心的人，希望找到一個和我一樣的人。', 'wish_gender': 'male', 'gender': 'female', 'mate': '我心目中的理想伴侶是一個陽光、幽默、有責任心的人。他應該是一個熱愛生活，積極向上的人。他應該也是一個有責任心的人，能夠照顧好我和家庭。'}
```
## Ref
- 


20231217-ntu-matches

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Ref
- https://www.kaggle.com/code/alanhc/gemini-api-starter-notebook
- https://makersuite.google.com/app/library

</body>
</html>



## Ref
- https://www.kaggle.com/code/alanhc/gemini-api-starter-notebook
- https://makersuite.google.com/app/library

20231217-google-gemini

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Ref
- https://learn.microsoft.com/en-us/azure/databricks/pandas/pyspark-pandas-conversion
- https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PySpark_Cheat_Sheet_Python.pdf
- [Pandas vs PySpark..!](https://medium.com/geekculture/pandas-vs-pyspark-fe110c266e5c)
- [Data Processing: Pandas vs PySpark vs Polars](https://medium.com/@benpinner1997/data-processing-pandas-vs-pyspark-vs-polars-fc1cdcb28725)
- [Polars vs. pandas: What’s the Difference?](https://blog.jetbrains.com/dataspell/2023/08/polars-vs-pandas-what-s-the-difference/)
-

</body>
</html>



## Ref
- https://learn.microsoft.com/en-us/azure/databricks/pandas/pyspark-pandas-conversion
-  https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PySpark_Cheat_Sheet_Python.pdf
- [Pandas vs PySpark..!](https://medium.com/geekculture/pandas-vs-pyspark-fe110c266e5c)
- [Data Processing: Pandas vs PySpark vs Polars](https://medium.com/@benpinner1997/data-processing-pandas-vs-pyspark-vs-polars-fc1cdcb28725)
- [Polars vs. pandas: What’s the Difference?](https://blog.jetbrains.com/dataspell/2023/08/polars-vs-pandas-what-s-the-difference/)
- 

20231213-pyspark

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## medium
https://github.com/dtesler/medium-to-markdown
## Blogger
https://codebeautify.org/html-to-markdown
https://github.com/cheshrkat/blogger-archive-converter
## Ref

</body>
</html>


## medium
https://github.com/dtesler/medium-to-markdown
## Blogger
https://codebeautify.org/html-to-markdown
https://github.com/cheshrkat/blogger-archive-converter

## Ref


20231210-convert-post-markdown

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Docker
`docker search tensorflow`
`docker pull jupyter/tensorflow-notebook`
`docker run jupyter/tensorflow-notebook -p 8000:8888`
- -p out:in
Transformers
`from transformers import pipeline`
`pip install transformers`
## docker compose
```yaml
services:
transformers-notebook:
build: ./Dockerfile | image: jupyter/tensorflow-notebook
- 8000:8888
environment
- JUPYTER_TOKEN=token
volumes
- ./:/home/{user}
```
```Dockerfile
FROM jupyter/tensorflow-notebook
USER $NB_UID //user | USER root
RUN pip install --upgrade pip &#x26;&#x26;\
pip install transformers &#x26;&#x26; \
pip install pysrt &#x26;&#x26; \
fix-permissions "/home/${NB_USER}"
COPY file.srt Translate.ipynb ./
```
`docker-compose up`
`from transformers import pipeline`
`pip install transformers`
```python
translator = pipeline("translation_en_to_fr")
fr = translator("Hi there I'm Alan!")
```
pysrt 讀取字幕檔
`docker images`
`docker image tag OOOOOOO-transformers-notebook:latest {user}/srt-translator:1.0`
`docker push {user}/srt-translator:1.0`
`docker container prune`
`docker rmi {user}/srt-translator:1.0`
`docker run -p 5000:8888 {user}/srt-translator:1.0`
## Ref
- https://youtu.be/-l7YocEQtA0?si=GOfJLcaa5CL8NBaX

</body>
</html>


## Docker
`docker search tensorflow`
`docker pull jupyter/tensorflow-notebook`
`docker run jupyter/tensorflow-notebook -p 8000:8888`
- -p out:in

Transformers
`from transformers import pipeline`
`pip install transformers`
## docker compose
```yaml
services:
	transformers-notebook:
			build: ./Dockerfile | image: jupyter/tensorflow-notebook
				- 8000:8888
			environment
				- JUPYTER_TOKEN=token
			volumes
				- ./:/home/{user}
```

```Dockerfile
FROM jupyter/tensorflow-notebook
USER $NB_UID   //user | USER root
RUN pip install --upgrade pip &&\
	pip install transformers && \
	pip install pysrt && \
	fix-permissions "/home/${NB_USER}"
COPY file.srt Translate.ipynb ./

```

`docker-compose up`

`from transformers import pipeline`
`pip install transformers`
```python
translator = pipeline("translation_en_to_fr")
fr = translator("Hi there I'm Alan!")

 ```
 pysrt 讀取字幕檔

`docker images`

`docker image tag OOOOOOO-transformers-notebook:latest {user}/srt-translator:1.0`

`docker push {user}/srt-translator:1.0`

`docker container prune`
`docker rmi {user}/srt-translator:1.0`

`docker run -p 5000:8888 {user}/srt-translator:1.0`

## Ref
- https://youtu.be/-l7YocEQtA0?si=GOfJLcaa5CL8NBaX


20231209-docker-machine-learning

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

AI 音樂風格轉換太酷了
[Frank Sinatra - A Cruel Angel's Thesis](https://youtu.be/IM_EDpvY4vA?si=jrhDvSM1wWdcnZgG)
[Careless Whisper - Frank Sinatra AI Cover](https://youtu.be/_Y543NEiR5w?si=uLzQnCi-u3sp-ZSV)
## Ref

</body>
</html>


AI 音樂風格轉換太酷了
[Frank Sinatra - A Cruel Angel's Thesis](https://youtu.be/IM_EDpvY4vA?si=jrhDvSM1wWdcnZgG)
[Careless Whisper - Frank Sinatra AI Cover](https://youtu.be/_Y543NEiR5w?si=uLzQnCi-u3sp-ZSV)
## Ref


20231209-AI-music-example

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## 1. 使用 twitter API
1. 先去 [dashboard]( https://developer.twitter.com/en/portal/dashboard )建立App，並在User authentication settings > 設定
![](https://i.imgur.com/zzfs8d9.png)
2. 複製 Keys and tokens
![](https://i.imgur.com/ifnkpvQ.png)
## 建立tweet
```python
import tweepy
bearer_token = ""
consumer_key = ""
consumer_secret = ""
access_token = ""
access_token_secret = ""
client = tweepy.Client(
consumer_key=consumer_key, consumer_secret=consumer_secret,
access_token=access_token, access_token_secret=access_token_secret
)
response = client.create_tweet(
text="This Tweet was Tweeted using Tweepy and Twitter API v2!"
)
print(f"https://twitter.com/user/status/{response.data['id']}")
# return
# Response(data={'edit_history_tweet_ids': ['1730651881314414696'], 'id': '1730651881314414696', 'text': 'This Tweet was Tweeted using Tweepy and Twitter API v2!1'}, includes={}, errors=[], meta={})
# get me
response = client.get_me()
response
```
免費版只有幾個 API
https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api
## 2. 使用 seleium
```python
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://twitter.com/i/flow/login')
username = driver.find_element(By.TAG_NAME, 'input')
username.send_keys("")
all_btn = driver.find_elements(By.XPATH, "//div[@role='button']")
all_btn[-2].click()
password = driver.find_element(By.XPATH, '//input[@type="password"]')
password.send_keys("")
all_btn = driver.find_elements(By.XPATH, "//div[@role='button']")
all_btn[-1].click()
keyword = "cat"
driver.get(f"https://twitter.com/search?q={keyword}&#x26;src=typed_query")
retweet = driver.find_elements(By.XPATH, "//div[@data-testid='retweet']")
retweet[0].click()
quote_tweet = driver.find_elements(By.XPATH, "//a[@role='menuitem']")
quote_tweet[0].click()
quote = driver.find_element(By.XPATH, "//div[contains(@class, 'public-DraftStyleDefault-block')]")
quote.send_keys("OMG!")
tweet = driver.find_element(By.XPATH, "//div[@data-testid='tweetButton']")
tweet.click()
driver.execute_script("window.scrollTo(0,document.body.scrollHeight);")
```
## Ref
https://developer.twitter.com/en/portal/dashboard
https://github.com/tweepy/tweepy/blob/master/examples/API_v2/create_tweet.py

</body>
</html>


## 1. 使用 twitter API
1. 先去 [dashboard]( https://developer.twitter.com/en/portal/dashboard )建立App，並在User authentication settings > 設定
![](https://i.imgur.com/zzfs8d9.png)
2. 複製 Keys and tokens 
![](https://i.imgur.com/ifnkpvQ.png)


## 建立tweet
```python
import tweepy

bearer_token = ""


consumer_key = ""

consumer_secret = ""

  
access_token = ""

access_token_secret = ""


client = tweepy.Client(

consumer_key=consumer_key, consumer_secret=consumer_secret,

access_token=access_token, access_token_secret=access_token_secret

)

response = client.create_tweet(

text="This Tweet was Tweeted using Tweepy and Twitter API v2!"

)
print(f"https://twitter.com/user/status/{response.data['id']}")
# return
# Response(data={'edit_history_tweet_ids': ['1730651881314414696'], 'id': '1730651881314414696', 'text': 'This Tweet was Tweeted using Tweepy and Twitter API v2!1'}, includes={}, errors=[], meta={})
  
# get me
response = client.get_me()

response
```
免費版只有幾個 API
https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api
## 2. 使用 seleium
```python
  

from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://twitter.com/i/flow/login')

username = driver.find_element(By.TAG_NAME, 'input')
username.send_keys("")

all_btn = driver.find_elements(By.XPATH, "//div[@role='button']")
all_btn[-2].click()

password = driver.find_element(By.XPATH, '//input[@type="password"]')

password.send_keys("")

all_btn = driver.find_elements(By.XPATH, "//div[@role='button']")
all_btn[-1].click()

keyword = "cat"
driver.get(f"https://twitter.com/search?q={keyword}&src=typed_query")

retweet = driver.find_elements(By.XPATH, "//div[@data-testid='retweet']")
retweet[0].click()

quote_tweet = driver.find_elements(By.XPATH, "//a[@role='menuitem']")

quote_tweet[0].click()
  

quote = driver.find_element(By.XPATH, "//div[contains(@class, 'public-DraftStyleDefault-block')]")

quote.send_keys("OMG!")

tweet = driver.find_element(By.XPATH, "//div[@data-testid='tweetButton']")

tweet.click()

driver.execute_script("window.scrollTo(0,document.body.scrollHeight);")

```
## Ref
https://developer.twitter.com/en/portal/dashboard
https://github.com/tweepy/tweepy/blob/master/examples/API_v2/create_tweet.py

20231202-twitter-bot

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Advance RAG Pipeline
![](https://i.imgur.com/8shM4tw.png)
![](https://i.imgur.com/EB5Xooa.png)
![](https://i.imgur.com/2RalCzF.png)
![](https://i.imgur.com/pMnkW5A.png)
![](https://i.imgur.com/ecL5SPj.png)
## RAG Triad of metrics
![](https://i.imgur.com/CSDiw5n.png)
![](https://i.imgur.com/VPFQWYN.png)
![](https://i.imgur.com/DOR4AMd.png)
![](https://i.imgur.com/0Qp6uFU.png)
![](https://i.imgur.com/mFGAbeG.png)
看到13:01
## Ref
- https://learn.deeplearning.ai/building-evaluating-advanced-rag

</body>
</html>


## Advance RAG Pipeline
![](https://i.imgur.com/8shM4tw.png)

![](https://i.imgur.com/EB5Xooa.png)
![](https://i.imgur.com/2RalCzF.png)

![](https://i.imgur.com/pMnkW5A.png)

![](https://i.imgur.com/ecL5SPj.png)
## RAG Triad of metrics
![](https://i.imgur.com/CSDiw5n.png)


![](https://i.imgur.com/VPFQWYN.png)
![](https://i.imgur.com/DOR4AMd.png)
![](https://i.imgur.com/0Qp6uFU.png)

![](https://i.imgur.com/mFGAbeG.png)
看到13:01

## Ref
- https://learn.deeplearning.ai/building-evaluating-advanced-rag

20231130-Building-and-Evaluating-Advanced-RAG (未完)

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## Ref
https://www.youtube.com/watch?v=y9k-U9AuDeM
![](https://i.imgur.com/RU3QUqs.png)

</body>
</html>



## Ref
https://www.youtube.com/watch?v=y9k-U9AuDeM
![](https://i.imgur.com/RU3QUqs.png)



Should You Use Open Source Large Language Models?

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

# 方式一 使用 GUI
## run service
e.g. `docker run -p 80:80 -d nginx`
## cloudflare 設定
https://dash.cloudflare.com/
dash > Zerotrust
![](https://i.imgur.com/C3QMcBJ.png)
![](https://i.imgur.com/h1tR17V.png)
![](https://i.imgur.com/DtY5i1w.png)
## 機器下載
註：windows 需要使用msi，用wsl 使用 docker會有問題
![](https://i.imgur.com/BVdQu81.png)
## 設定service位址
![](https://i.imgur.com/3swAtw9.png)
## 結果
訪問 前一個步驟設定的網址，成功🎉
![](https://i.imgur.com/QPcQh08.png)
# 方式2 使用 docker compose
TOKEN 為
```yaml
version: "3.9"
services:
nginx:
image: nginx:latest
expose:
- 80
cloudflared:
image: cloudflare/cloudflared:latest
command: tunnel run
environment:
- TUNNEL_TOKEN={TOKEN}
```
public host 設定
`docker-compose up -d`
## Ref

</body>
</html>


# 方式一 使用 GUI
## run service
e.g. `docker run -p 80:80 -d nginx`
## cloudflare 設定
https://dash.cloudflare.com/
dash > Zerotrust
![](https://i.imgur.com/C3QMcBJ.png)
![](https://i.imgur.com/h1tR17V.png)
![](https://i.imgur.com/DtY5i1w.png)
## 機器下載
註：windows 需要使用msi，用wsl 使用 docker會有問題
![](https://i.imgur.com/BVdQu81.png)
## 設定service位址

![](https://i.imgur.com/3swAtw9.png)

## 結果
訪問 前一個步驟設定的網址，成功🎉
![](https://i.imgur.com/QPcQh08.png)

# 方式2 使用 docker compose
TOKEN 為
```yaml
version: "3.9"  
services:  
nginx:  
image: nginx:latest  
expose:  
- 80  
cloudflared:  
image: cloudflare/cloudflared:latest  
command: tunnel run  
environment:  
- TUNNEL_TOKEN={TOKEN}
```
public host 設定 
`docker-compose up -d`
## Ref


20231125-cloudflare-tunnel

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

vercel token: https://vercel.com/guides/how-do-i-use-a-vercel-api-access-token
npx create-next-app nextjs-terraform-demo
main.tf
```terraform
terraform {
required_providers {
vercel = {
source = "vercel/vercel"
version = "~> 0.3"
}
}
}
```
`terraform init`
```terraform
terraform {
required_providers {
vercel = {
source = "vercel/vercel"
version = "~> 0.3"
}
}
}
resource "vercel_project" "example" {
name = "terraform-test-project"
framework = "nextjs"
git_repository = {
type = "github"
repo = "alanhc/nextjs-terraform-demo"
}
}
```
terraform apply
## Ref
- https://vercel.com/guides/integrating-terraform-with-vercel
- https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli
- https://vercel.com/guides/how-do-i-use-a-vercel-api-access-token

</body>
</html>


vercel token: https://vercel.com/guides/how-do-i-use-a-vercel-api-access-token
npx create-next-app nextjs-terraform-demo
main.tf
```terraform
terraform {
  required_providers {
    vercel = {
      source = "vercel/vercel"
      version = "~> 0.3"
    }
  }
}
```
`terraform init`
```terraform
terraform {
  required_providers {
    vercel = {
      source = "vercel/vercel"
      version = "~> 0.3"
    }
  }
}
resource "vercel_project" "example" {
  name      = "terraform-test-project"
  framework = "nextjs"
  git_repository = {
    type = "github"
    repo = "alanhc/nextjs-terraform-demo"
  }
}
```
terraform apply
## Ref
- https://vercel.com/guides/integrating-terraform-with-vercel
- https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli
- https://vercel.com/guides/how-do-i-use-a-vercel-api-access-token

20231124-terraform-vercel

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

RAG -> vector database
![](https://i.imgur.com/EDn3CJo.png)
## how-to-obtain-vector-representations-of-data
![](https://i.imgur.com/OdWVtRa.png)
![](https://i.imgur.com/pW8RpmW.png)
![[L1_Embeddings]]
## search-for-similar-vectors
![](https://i.imgur.com/sHpnu1f.png)
![](https://i.imgur.com/tXyrNpE.png)
![[L2_kNN (1)]]
## approximate-nearest-neighbours
![](https://i.imgur.com/dhCmV9R.png)
![](https://i.imgur.com/PuuhCfX.png)
![](https://i.imgur.com/zM1uhkl.png)
![](https://i.imgur.com/HIbOnKD.png)
![](https://i.imgur.com/ARkaKNP.png)
![](https://i.imgur.com/UXKVBIP.png)
![](https://i.imgur.com/2dzDBMZ.png)
![](https://i.imgur.com/fBe2Jwc.png)
![](https://i.imgur.com/AkDazPU.png)
![](https://i.imgur.com/2Ckgxvl.png)
![[L3_approximate_nearest_neighbours]]
## vector-databases
![[L4_Objects_Vectors]]
## sparse,-dense,-and-hybrid-search
![](https://i.imgur.com/X0Sdkgn.png)
![](https://i.imgur.com/SD5AbUM.png)
![](https://i.imgur.com/lw9WLkZ.png)
![](https://i.imgur.com/VuWuQfv.png)
![](https://i.imgur.com/Rd2FsOC.png)
![[L5_HybridSearch]]
## application---multilingual-search
![](https://i.imgur.com/uXHEwdA.png)
![](https://i.imgur.com/y3e7Uoy.png)
![](https://i.imgur.com/a91aqXe.png)
![](https://i.imgur.com/QyhQ2DD.png)
![](https://i.imgur.com/0JVqpXl.png)
![[L6_Multilingual_RAG_Search]]
## Ref
- https://learn.deeplearning.ai/vector-databases-embeddings-applications

</body>
</html>


RAG -> vector database
 
![](https://i.imgur.com/EDn3CJo.png)

## how-to-obtain-vector-representations-of-data
![](https://i.imgur.com/OdWVtRa.png)
![](https://i.imgur.com/pW8RpmW.png)

![[L1_Embeddings]]
## search-for-similar-vectors

![](https://i.imgur.com/sHpnu1f.png)

![](https://i.imgur.com/tXyrNpE.png)

![[L2_kNN (1)]]
## approximate-nearest-neighbours

![](https://i.imgur.com/dhCmV9R.png)

![](https://i.imgur.com/PuuhCfX.png)

![](https://i.imgur.com/zM1uhkl.png)

![](https://i.imgur.com/HIbOnKD.png)

![](https://i.imgur.com/ARkaKNP.png)

![](https://i.imgur.com/UXKVBIP.png)

![](https://i.imgur.com/2dzDBMZ.png)

![](https://i.imgur.com/fBe2Jwc.png)

![](https://i.imgur.com/AkDazPU.png)

![](https://i.imgur.com/2Ckgxvl.png)
![[L3_approximate_nearest_neighbours]]

## vector-databases
![[L4_Objects_Vectors]]
## sparse,-dense,-and-hybrid-search

![](https://i.imgur.com/X0Sdkgn.png)


![](https://i.imgur.com/SD5AbUM.png)

![](https://i.imgur.com/lw9WLkZ.png)

![](https://i.imgur.com/VuWuQfv.png)

![](https://i.imgur.com/Rd2FsOC.png)
![[L5_HybridSearch]]
## application---multilingual-search
![](https://i.imgur.com/uXHEwdA.png)

![](https://i.imgur.com/y3e7Uoy.png)
![](https://i.imgur.com/a91aqXe.png)
![](https://i.imgur.com/QyhQ2DD.png)
![](https://i.imgur.com/0JVqpXl.png)
![[L6_Multilingual_RAG_Search]]

## Ref
- https://learn.deeplearning.ai/vector-databases-embeddings-applications

20231116-Vector-Databases-from-Embeddings-to-Applications

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

16:52
twich 創辦人（Michael Seibel）
![](https://i.imgur.com/dLhPAHZ.png)
## 1. 從使用者學習
![](https://i.imgur.com/Rxqo8ls.png)
![](https://i.imgur.com/1t93zk7.png)
不要害怕失去客戶，因為才剛開始
Fake steve jobs
不要執著打造完美產品
e.g. 第一次的iphone 沒有app store的iphone 都是 經過時間迭代的產品
![](https://i.imgur.com/efBCb1M.png)
第一代 Airbnb
![](https://i.imgur.com/VIuNtyJ.png)
![](https://i.imgur.com/nqHbCFI.png)
![](https://i.imgur.com/omzsTHQ.png)
![](https://i.imgur.com/4zBBiLk.png)
解決問題是你的責任，不是客戶的
從市場中學習
一個MVP是最快從客戶學習的方法
![](https://i.imgur.com/uYLXQyJ.png)
## Ref
- https://youtu.be/QRZ_l7cVzzU?si=oGWmErG3cp8hxwtq

</body>
</html>


16:52
twich 創辦人（Michael Seibel）

![](https://i.imgur.com/dLhPAHZ.png)
## 1. 從使用者學習

![](https://i.imgur.com/Rxqo8ls.png)

![](https://i.imgur.com/1t93zk7.png)
不要害怕失去客戶，因為才剛開始
Fake steve jobs
不要執著打造完美產品
e.g. 第一次的iphone 沒有app store的iphone 都是 經過時間迭代的產品

![](https://i.imgur.com/efBCb1M.png)
第一代 Airbnb 
![](https://i.imgur.com/VIuNtyJ.png)

![](https://i.imgur.com/nqHbCFI.png)

![](https://i.imgur.com/omzsTHQ.png)
![](https://i.imgur.com/4zBBiLk.png)
解決問題是你的責任，不是客戶的
從市場中學習
一個MVP是最快從客戶學習的方法

![](https://i.imgur.com/uYLXQyJ.png)

## Ref
- https://youtu.be/QRZ_l7cVzzU?si=oGWmErG3cp8hxwtq


20231115-How-to-Build-An-MVP-Startup-School

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

![](https://i.imgur.com/0k8rQsV.png)
problem?
關鍵字搜尋
![](https://i.imgur.com/mTMj2P7.png)
![](https://i.imgur.com/9W5ClYB.png)
vector embeding
![](https://i.imgur.com/3v1g4KG.png)
![](https://i.imgur.com/bs7knP1.png)
https://github.com/weaviate-tutorials/multimodal-workshop
可以直接使用openai 提供 或 self host
## 1.1
## 1.2
alpha: 1: 使用100%vector search 0.7:70% vector 0.3: keyword
## 2
![](https://i.imgur.com/30KnOPa.png)
![](https://i.imgur.com/XZVyg8O.png)
![](https://i.imgur.com/dRkv1Hw.png)
![](https://i.imgur.com/0y3PEE3.png)
![](https://i.imgur.com/yBShmFS.png)
![](https://i.imgur.com/bPdIbwa.png)
![](https://i.imgur.com/ugwLGJq.png)
## Ref
- https://www.youtube.com/watch?v=3WUobZryyok
- https://github.com/weaviate-tutorials/multimodal-workshop
- https://github.com/alanhc/multimodal-workshop
- https://docs.google.com/presentation/d/1sS-bxJ-k9EuESH2VhpwnybY3QyV_9FdxHLmZLCSpuSM/edit#slide=id.g29ae64ebf98_0_5

</body>
</html>





![](https://i.imgur.com/0k8rQsV.png)
problem?
關鍵字搜尋
![](https://i.imgur.com/mTMj2P7.png)

![](https://i.imgur.com/9W5ClYB.png)
vector embeding


![](https://i.imgur.com/3v1g4KG.png)


![](https://i.imgur.com/bs7knP1.png)


https://github.com/weaviate-tutorials/multimodal-workshop
可以直接使用openai 提供 或 self host
## 1.1
## 1.2
alpha: 1: 使用100%vector search  0.7:70% vector 0.3: keyword


## 2
![](https://i.imgur.com/30KnOPa.png)


![](https://i.imgur.com/XZVyg8O.png)

![](https://i.imgur.com/dRkv1Hw.png)

![](https://i.imgur.com/0y3PEE3.png)

![](https://i.imgur.com/yBShmFS.png)

![](https://i.imgur.com/bPdIbwa.png)



![](https://i.imgur.com/ugwLGJq.png)

## Ref
- https://www.youtube.com/watch?v=3WUobZryyok
- https://github.com/weaviate-tutorials/multimodal-workshop
- https://github.com/alanhc/multimodal-workshop
- https://docs.google.com/presentation/d/1sS-bxJ-k9EuESH2VhpwnybY3QyV_9FdxHLmZLCSpuSM/edit#slide=id.g29ae64ebf98_0_5


20211115-LIVEBuilding-Multi-Modal-Search-with-Vector-Databases

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## upload api
```js
import { connectToDb, fileExists } from "@/lib/mongodb";
import { NextResponse } from "next/server";
import { Readable } from "stream";
import formidable, { errors as formidableErrors } from 'formidable';
var fs = require('fs');
var md5 = require('md5');
export const config = {
api: {
bodyParser: false,
}
};
type ProcessedFiles = Array&#x3C;[string, File]>;
export default async function handler(req: any, res: any) {
const { bucket } = await connectToDb();
// get the form data
// const data = await req.formData();
// Access uploaded files directly using req.files
//const files = Array.from(req.files.entries());
let status = 200,
resultBody = { status: 'ok', message: 'Files were uploaded successfully' };
const form = formidable({ uploadDir: "/tmp" });
let fields;
let files;
try {
form.parse(req, async (err, fields, files) => {
console.log('fields:', fields);
console.log('files:', files);
for (const [key, value] of Object.entries(files)) {
const isFile = typeof value == "object";
if (isFile) {
let file: any = value[0]
let filename = file.originalFilename
let type = file.mimetype
let buffer = fs.readFileSync(file.filepath);
const stream = Readable.from(buffer);
const hash = md5(buffer)
const existing = await fileExists(hash);
if (existing) {
// If file already exists, let's skip it.
// If you want a different behavior such as override, modify this part.
continue;
}
const uploadStream = bucket.openUploadStream(filename, {
// make sure to add content type so that it will be easier to set later.
contentType: type,
metadata: {
hash:hash
}, //add your metadata here if any
});
// pipe the readable stream to a writeable stream to save it to the database
await stream.pipe(uploadStream);
res.status(200).json({
hash: hash
})
}
}
});
} catch (err: any) {
// example to check for a very specific error
if (err.code === formidableErrors.maxFieldsExceeded) {
}
console.error(err);
res.writeHead(err.httpCode || 400, { 'Content-Type': 'text/plain' });
res.end(String(err));
return;
}
return res.json({ success: true });
}
```
## response image
```js
import { MongoClient, ObjectId, GridFSBucket } from 'mongodb';
import clientPromise, { connectToDb } from "@/lib/mongodb";
export default async function handler(req:any, res:any) {
const client = await clientPromise;
const {hash} = req.query;
const db = client.db("fanstick");
const metadata:any = await db
.collection('media.files')
.find({"metadata.hash":hash})
.toArray()
console.log(metadata)
const {bucket}= await connectToDb()
res.writeHead(200, { 'Content-Type': metadata[0].contentType });
bucket.openDownloadStream(metadata[0]._id)
.on('data', (chunk) => {
res.write(chunk);
})
.on('end', () => {
res.end();
})
.on('error', (err) => {
res.status(500).json({ error: err.message });
});
// const file:any = await db
// .collection('media.chunks')
// .find({"files_id":metadata[0]._id})
// .toArray()
// console.log(file[0].data)
// res.setHeader('Content-Type', metadata[0].contentType)
// res.send(file[0].data)
}
```
## Frontend
```js
fetch(`/backend/upload/file`, {
method: "POST",
body: formData
}).then((res) => res.json())
.then(({hash}) => {
const image_url = `${process.env.Deploy_URL}/api/file/${hash}`
setImage(image_url)
form.setFieldValue(field.name,image_url)
})
```
## Ref
- https://github.com/gapon2401/upload-files-nextjs/blob/master/pages/api/upload.ts
- https://reacthustle.com/blog/how-to-upload-retrieve-images-to-mongodb-using-nextjs-13-app-router

</body>
</html>


## upload api
```js

import { connectToDb, fileExists } from "@/lib/mongodb";

import { NextResponse } from "next/server";

import { Readable } from "stream";

import formidable, { errors as formidableErrors } from 'formidable';

var fs = require('fs');

var md5 = require('md5');

  

export const config = {

api: {

bodyParser: false,

}

};

type ProcessedFiles = Array<[string, File]>;

export default async function handler(req: any, res: any) {

const { bucket } = await connectToDb();

// get the form data

// const data = await req.formData();

// Access uploaded files directly using req.files

//const files = Array.from(req.files.entries());

  

let status = 200,

resultBody = { status: 'ok', message: 'Files were uploaded successfully' };

  

const form = formidable({ uploadDir: "/tmp" });

let fields;

let files;

try {

form.parse(req, async (err, fields, files) => {

console.log('fields:', fields);

console.log('files:', files);

for (const [key, value] of Object.entries(files)) {

const isFile = typeof value == "object";

if (isFile) {

let file: any = value[0]

let filename = file.originalFilename

let type = file.mimetype

let buffer = fs.readFileSync(file.filepath);

const stream = Readable.from(buffer);

const hash = md5(buffer)

const existing = await fileExists(hash);

if (existing) {

// If file already exists, let's skip it.

// If you want a different behavior such as override, modify this part.

continue;

}

const uploadStream = bucket.openUploadStream(filename, {

// make sure to add content type so that it will be easier to set later.

contentType: type,

metadata: {

hash:hash

}, //add your metadata here if any

});

  

// pipe the readable stream to a writeable stream to save it to the database

await stream.pipe(uploadStream);

res.status(200).json({

hash: hash

})

}

  

}

});

  
  

} catch (err: any) {

// example to check for a very specific error

if (err.code === formidableErrors.maxFieldsExceeded) {

  

}

console.error(err);

res.writeHead(err.httpCode || 400, { 'Content-Type': 'text/plain' });

res.end(String(err));

return;

}

  

return res.json({ success: true });

}
```
## response image
```js
import { MongoClient, ObjectId, GridFSBucket } from 'mongodb';

import clientPromise, { connectToDb } from "@/lib/mongodb";


export default async function handler(req:any, res:any) {

const client = await clientPromise;

const {hash} = req.query;

const db = client.db("fanstick");

const metadata:any = await db

.collection('media.files')

.find({"metadata.hash":hash})

.toArray()

console.log(metadata)

const {bucket}= await connectToDb()

res.writeHead(200, { 'Content-Type': metadata[0].contentType });

bucket.openDownloadStream(metadata[0]._id)

.on('data', (chunk) => {

res.write(chunk);

})

.on('end', () => {

res.end();

})

.on('error', (err) => {

res.status(500).json({ error: err.message });

});

// const file:any = await db

// .collection('media.chunks')

// .find({"files_id":metadata[0]._id})

// .toArray()

// console.log(file[0].data)

// res.setHeader('Content-Type', metadata[0].contentType)

// res.send(file[0].data)

}
```
## Frontend
```js
fetch(`/backend/upload/file`, {

method: "POST",

body: formData

}).then((res) => res.json())

.then(({hash}) => {

const image_url = `${process.env.Deploy_URL}/api/file/${hash}`

setImage(image_url)

form.setFieldValue(field.name,image_url)

})
```
## Ref
- https://github.com/gapon2401/upload-files-nextjs/blob/master/pages/api/upload.ts
- https://reacthustle.com/blog/how-to-upload-retrieve-images-to-mongodb-using-nextjs-13-app-router

2021112-upload-large-file

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

![](https://i.imgur.com/E4SLakC.png)
![](https://i.imgur.com/NBeKimk.png)
![](https://i.imgur.com/m6eJNV9.png)
![](https://i.imgur.com/8eGCc4I.png)
![](https://i.imgur.com/fhbGaPQ.png)
![](https://i.imgur.com/Y16g0q5.png)
![](https://i.imgur.com/4NpQoXl.png)
![](https://i.imgur.com/nLljMpP.png)
![](https://i.imgur.com/MAIF5sp.png)
![](https://i.imgur.com/XvRU13y.png)
![](https://i.imgur.com/lkIkFbG.png)
![](https://i.imgur.com/IzW4c71.png)
![](https://i.imgur.com/WFF0Y5J.png)
## Ref
- https://www.youtube.com/watch?v=azBugJzmz-o

</body>
</html>



![](https://i.imgur.com/E4SLakC.png)
![](https://i.imgur.com/NBeKimk.png)

![](https://i.imgur.com/m6eJNV9.png)
![](https://i.imgur.com/8eGCc4I.png)

![](https://i.imgur.com/fhbGaPQ.png)
 
![](https://i.imgur.com/Y16g0q5.png)


![](https://i.imgur.com/4NpQoXl.png)
![](https://i.imgur.com/nLljMpP.png)
![](https://i.imgur.com/MAIF5sp.png)
![](https://i.imgur.com/XvRU13y.png)
![](https://i.imgur.com/lkIkFbG.png)
![](https://i.imgur.com/IzW4c71.png)
![](https://i.imgur.com/WFF0Y5J.png)

## Ref
- https://www.youtube.com/watch?v=azBugJzmz-o


20231110-生成式AI淺談圖像生成模型-Diffusion-Model-原理

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

![](https://i.imgur.com/U6OeFM7.png)
客製化GPT
https://openai.com/blog/introducing-gpts
串接dalle 自動生成頭像
![](https://i.imgur.com/oBxJeFw.png)
GPT: Generative Pre-trained Transformer
others Google Bard anthropic Claude
GPT 訓練：文字接龍 e.g. 台灣大->chatGPT-> 學
台灣大->chatGPT-> （機率）學10% 歌、6%
![](https://i.imgur.com/WxgJVLd.png)
https://platform.openai.com/docs/tokenizer
![](https://i.imgur.com/OaqzMkQ.png)
ChatGPT目前可以讀300頁pdt
![](https://i.imgur.com/LakJpvl.png)
![](https://i.imgur.com/N38im7J.png)
![](https://i.imgur.com/rADG2nT.png)
![](https://i.imgur.com/p5wvO2f.png)
![](https://i.imgur.com/LU8OEvz.png)
![](https://i.imgur.com/sAZAVkn.png)
![](https://i.imgur.com/jymlc5s.png)
![](https://i.imgur.com/td5xUOQ.png)
![](https://i.imgur.com/QUuYMwJ.png)
![](https://i.imgur.com/d9NR3Tm.png)
![](https://i.imgur.com/eD4pabX.png)
chatgpt可能跟 Instruct gpt有關
![](https://i.imgur.com/fnq016C.png)
![](https://i.imgur.com/XnJi3eQ.png)
![](https://i.imgur.com/QOxjDFZ.png)
![](https://i.imgur.com/kG7tiSR.png)
![](https://i.imgur.com/NFesrFO.png)
![](https://i.imgur.com/90J8vor.png)
![](https://i.imgur.com/R3EC4JH.png)
![](https://i.imgur.com/E2PHJyD.png)
![](https://i.imgur.com/8NPELGC.png)
gpt 3.5 人類去調整
![](https://i.imgur.com/Gx0UFI1.png)
![](https://i.imgur.com/UUoGWE3.png)
## 激發語言模型
1. 需求講清楚
1. e.g. 擴寫、文法
2. 提供資訊給chatgpt
3. 提供範例
4. 想一想 chain of thiught
5. 找出神奇的prompt
6. 上傳檔案
1. 製作投影片、外掛程式
7. 使用其它工具
8. 拆解任務
9. 自主進行規劃
10. 模型自我反省
11. 跟真實環境互動
![](https://i.imgur.com/8fMEfuR.png)
![](https://i.imgur.com/HSVx09i.png)
![](https://i.imgur.com/9hq7EFO.png)
![](https://i.imgur.com/VG2Sqqy.png)
![](https://i.imgur.com/nooGFgj.png)
![](https://i.imgur.com/ABZCSi6.png)
![](https://i.imgur.com/OtG7Afm.png)
![](https://i.imgur.com/2tef2w9.png)
![](https://i.imgur.com/zq78GhS.png)
![](https://i.imgur.com/s0IVFSc.png)
GPT 4 以後才有反省能力
![](https://i.imgur.com/ntmOaPw.png)
![](https://i.imgur.com/ePCj9UD.png)
![](https://i.imgur.com/L1Dw49l.png)
![](https://i.imgur.com/jgjkTj0.png)
![](https://i.imgur.com/nK4hG2R.png)
![](https://i.imgur.com/KXFhfgh.png)
## Ref
- https://www.youtube.com/watch?v=wG8-IUtqu-s&#x26;t=365s

</body>
</html>



![](https://i.imgur.com/U6OeFM7.png)
客製化GPT
https://openai.com/blog/introducing-gpts
串接dalle 自動生成頭像

![](https://i.imgur.com/oBxJeFw.png)
GPT: Generative Pre-trained Transformer
others Google Bard anthropic Claude
GPT 訓練：文字接龍 e.g. 台灣大->chatGPT-> 學
 台灣大->chatGPT-> （機率）學10% 歌、6%
![](https://i.imgur.com/WxgJVLd.png)
https://platform.openai.com/docs/tokenizer
![](https://i.imgur.com/OaqzMkQ.png)

ChatGPT目前可以讀300頁pdt

![](https://i.imgur.com/LakJpvl.png)


![](https://i.imgur.com/N38im7J.png)

![](https://i.imgur.com/rADG2nT.png)

![](https://i.imgur.com/p5wvO2f.png)

![](https://i.imgur.com/LU8OEvz.png)

![](https://i.imgur.com/sAZAVkn.png)

![](https://i.imgur.com/jymlc5s.png)

![](https://i.imgur.com/td5xUOQ.png)

![](https://i.imgur.com/QUuYMwJ.png)


![](https://i.imgur.com/d9NR3Tm.png)

![](https://i.imgur.com/eD4pabX.png)
chatgpt可能跟 Instruct gpt有關

![](https://i.imgur.com/fnq016C.png)

![](https://i.imgur.com/XnJi3eQ.png)

![](https://i.imgur.com/QOxjDFZ.png)

![](https://i.imgur.com/kG7tiSR.png)

![](https://i.imgur.com/NFesrFO.png)

![](https://i.imgur.com/90J8vor.png)

![](https://i.imgur.com/R3EC4JH.png)

![](https://i.imgur.com/E2PHJyD.png)

![](https://i.imgur.com/8NPELGC.png)
gpt 3.5 人類去調整
![](https://i.imgur.com/Gx0UFI1.png)

![](https://i.imgur.com/UUoGWE3.png)
## 激發語言模型
1. 需求講清楚
	1. e.g. 擴寫、文法
2. 提供資訊給chatgpt
3.  提供範例
4. 想一想 chain of thiught
5. 找出神奇的prompt
6. 上傳檔案
	1. 製作投影片、外掛程式
7. 使用其它工具
8. 拆解任務
9. 自主進行規劃
10. 模型自我反省
11. 跟真實環境互動

![](https://i.imgur.com/8fMEfuR.png)
![](https://i.imgur.com/HSVx09i.png)

![](https://i.imgur.com/9hq7EFO.png)

![](https://i.imgur.com/VG2Sqqy.png)


![](https://i.imgur.com/nooGFgj.png)

![](https://i.imgur.com/ABZCSi6.png)
![](https://i.imgur.com/OtG7Afm.png)
![](https://i.imgur.com/2tef2w9.png)

![](https://i.imgur.com/zq78GhS.png)

![](https://i.imgur.com/s0IVFSc.png)
GPT 4 以後才有反省能力
![](https://i.imgur.com/ntmOaPw.png)

![](https://i.imgur.com/ePCj9UD.png)


![](https://i.imgur.com/L1Dw49l.png)

![](https://i.imgur.com/jgjkTj0.png)

![](https://i.imgur.com/nK4hG2R.png)

![](https://i.imgur.com/KXFhfgh.png)

## Ref
- https://www.youtube.com/watch?v=wG8-IUtqu-s&t=365s

80分鐘快速了解大型語言模型-5-30 有咒術迴戰雷

20231108-How-Diffusion-Models-Work

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

免費圖片上傳api
```js
const onDrop = useCallback((acceptedFiles: any) => {
acceptedFiles.forEach((file: any) => {
const reader:any = new FileReader()
reader.onabort = () => console.log('file reading was aborted')
reader.onerror = () => console.log('file reading has failed')
reader.onload = async () => {
const base64String:any = reader.result.split(',')[1];
console.log(base64String);
let data = new FormData()
data.append("image", base64String)
fetch(`https://api.imgbb.com/1/upload?expiration=600&#x26;key=${api_key}`, {
method: "POST",
body:data
}).then((res)=>console.log(res))
}
reader.readAsDataURL(file)
})
}, [])
```
## Ref
- https://api.imgbb.com/

</body>
</html>


免費圖片上傳api
```js
const onDrop = useCallback((acceptedFiles: any) => {
acceptedFiles.forEach((file: any) => {

const reader:any = new FileReader()

reader.onabort = () => console.log('file reading was aborted')

reader.onerror = () => console.log('file reading has failed')

reader.onload = async () => {

  

const base64String:any = reader.result.split(',')[1];

console.log(base64String);

let data = new FormData()

data.append("image", base64String)

  

fetch(`https://api.imgbb.com/1/upload?expiration=600&key=${api_key}`, {

method: "POST",

body:data

}).then((res)=>console.log(res))

}

reader.readAsDataURL(file)

})

  

}, [])
```
## Ref
- https://api.imgbb.com/

20231107-imgbb

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

很好做prototype的前端網站
## Ref
- https://mui.com/toolpad/getting-started/first-app/

</body>
</html>


很好做prototype的前端網站
## Ref
- https://mui.com/toolpad/getting-started/first-app/

20231107-MUI-toolpad

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

[Ignite CLI](https://ignite.com/cli): [它是Cosmos SDK](https://github.com/cosmos/cosmos-sdk)的開發人員友善介面
https://docs.ignite.com/guide
建立區塊鏈
ignite scaffold chain mars
cd mars
ignite chain serve
ignite scaffold chain hello
查詢
nft marketplace
https://www.stargaze.zone/marketplace
## Ref
-

</body>
</html>



[Ignite CLI](https://ignite.com/cli): [它是Cosmos SDK](https://github.com/cosmos/cosmos-sdk)的開發人員友善介面
https://docs.ignite.com/guide

建立區塊鏈
ignite scaffold chain mars

cd mars

ignite chain serve

ignite scaffold chain hello

查詢


nft marketplace
https://www.stargaze.zone/marketplace
## Ref
- 

20231104-learn-cosmos

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

### c++
```c++
#include
<div>
  using namespace std;
  struct Node {
  int data;
  Node* next;
  };
  Node* head = nullptr;
  void insertNode(int data)
  {
  Node* newNode = new Node();
  newNode->data = data;
  newNode->next = nullptr;
  //head沒有東西
  if (head == nullptr) {
  head = newNode;
  return;
  }
  //head有東西
  Node* now = head;
  //找到 ->next 是空的接上去
  while (now->next!=nullptr) {
  now = now->next;
  }
  now->next = newNode;
  }
  void deleteNode(int data)
  {
  if (head==nullptr) return;
  //找到要刪除的節點
  Node* to_delete = head;
  Node* prev = nullptr;
  while (to_delete->data != data) {
  prev = to_delete;
  to_delete = to_delete->next;
  }
  if (to_delete!=nullptr) {
  if(prev == nullptr) head = to_delete->next;
  else prev->next = to_delete->next;
  delete to_delete;
  }
  }
  void printList()
  {
  Node* now = head;
  while (now!=nullptr) {
  cout &#x3C;&#x3C; now->data &#x3C;&#x3C; " ";
  now = now-> next;
  }
  cout &#x3C;&#x3C; endl;
  }
  int main() {
  insertNode(1);
  insertNode(2);
  insertNode(3);
  insertNode(4);
  insertNode(5);
  printList();
  // 刪除首節點
  deleteNode(1);
  printList();
  // 刪除中間
  deleteNode(3);
  printList();
  return 0;
  }
  ```
  ## Ref
</div>

</body>
</html>



### c++
```c++
#include <iostream>
using namespace std;

struct Node {
    int data;
    Node* next;
};

Node* head = nullptr;
void insertNode(int data)
{
    Node* newNode = new Node();
    newNode->data = data;
    newNode->next = nullptr;

    //head沒有東西
    if (head == nullptr) {
        head = newNode;
        return; 
    }
    //head有東西
    Node* now = head;
    //找到 ->next 是空的接上去
    while (now->next!=nullptr) {
        now = now->next;
    }
    now->next = newNode;
}
void deleteNode(int data)
{
    if (head==nullptr) return; 
    //找到要刪除的節點
    Node* to_delete = head;
    Node* prev = nullptr;
    while (to_delete->data != data) {
        prev = to_delete;
        to_delete = to_delete->next;
    }
    if (to_delete!=nullptr) {
        if(prev == nullptr) head = to_delete->next;
        else prev->next = to_delete->next;

        delete to_delete;
    }   
}
void printList()
{
    Node* now = head;
    while (now!=nullptr) {
        cout << now->data << " ";
        now = now-> next;
    }
    cout << endl;
}
int main() {
    insertNode(1);
    insertNode(2);
    insertNode(3);
    insertNode(4);
    insertNode(5);
    printList();
    // 刪除首節點
    deleteNode(1);
    printList();
    // 刪除中間
    deleteNode(3);
    printList();
    
    return 0;
}
```
## Ref


20231103-linklist

20231103-Finetuning-Large-Language-Models

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## What is Generative AI
![](https://i.imgur.com/Q877q0W.png)
![](https://i.imgur.com/8DpeAAb.png)
![](https://i.imgur.com/FhLUDBG.png)
- Supervised learning (labeling things)
![](https://i.imgur.com/fFv1p2L.png)
- 2010 - 2020: Large scale supervised learning
- LLM
- How? supervised learning (A->B) 重複預測下一個word
![](https://i.imgur.com/d3JGgOK.png)
- 例子
- 寫作：rewrite for clarity
- 閱讀：有沒有在抱怨、情緒分析
- 聊天：聊天機器人
- web search or LLM？
![](https://i.imgur.com/PIehfwc.png)
- LLM可能會錯，但回答比較精簡
- web search有時會得到比較好的答案，但要花時間找到你要的資訊
## Generative AI Applications
![](https://i.imgur.com/ubs4shO.png)
![](https://i.imgur.com/ZblmQDs.png)
![](https://i.imgur.com/F1oNomh.png)
- setup
```python
import openai
import os
openai.api_key = os.getenv("OPENAI_API_KEY")
def llm_response(prompt):
response = openai.ChatCompletion.create(
model='gpt-3.5-turbo',
messages=[{'role':'user','content':prompt}],
temperature=0
)
return response.choices[0].message['content']
```
- classify
```python
prompt = '''
Classify the following review
as having either a positive or
negative sentiment:
The banana pudding was really tasty!
'''
response = llm_response(prompt)
print(response)
```
```python
all_reviews = [
'The mochi is excellent!',
'Best soup dumplings I have ever eaten.',
'Not worth the 3 month wait for a reservation.',
'The colorful tablecloths made me smile!',
'The pasta was cold.'
]
all_reviews
classifications = []
for review in all_reviews:
prompt = f'''
Classify the following review
as having either a positive or
negative sentiment. State your answer
as a single word, either "positive" or
"negative":
{review}
'''
response = llm_response(prompt)
classifications.append(response)
classifications
```
![](https://i.imgur.com/MoG5DoG.png)
![](https://i.imgur.com/VOdC3dn.png)
![](https://i.imgur.com/zbDvhYJ.png)
![](https://i.imgur.com/ranyh2o.png)
![](https://i.imgur.com/BSS1Rsx.png)
### Advance technologies: Beyond prompting
![](https://i.imgur.com/EeOJZiM.png)
![](https://i.imgur.com/kkk2PLP.png)
![](https://i.imgur.com/Zx9Yun9.png)
![](https://i.imgur.com/EoqwkuW.png)
![](https://i.imgur.com/Ysjf6Bj.png)
![](https://i.imgur.com/cBEfLR0.png)
![](https://i.imgur.com/wdyy3ov.png)
![](https://i.imgur.com/4uomkXV.png)
![](https://i.imgur.com/VTx5KBM.png)
![](https://i.imgur.com/HM0jMCS.png)
![](https://i.imgur.com/DVqNRPm.png)
![](https://i.imgur.com/dnGp4io.png)
![](https://i.imgur.com/qOxdVE0.png)
![](https://i.imgur.com/JiWy6ky.png)
![](https://i.imgur.com/g9sp9gj.png)
![](https://i.imgur.com/DYyC9xa.png)
![](https://i.imgur.com/RmNpxgS.png)
![](https://i.imgur.com/YVs5iOT.png)
![](https://i.imgur.com/kvfmIsI.png)
![](https://i.imgur.com/s5j8JfV.png)
![](https://i.imgur.com/It3c1bn.png)
![](https://i.imgur.com/TrH6LIS.png)
## Generative AI and Business
![](https://i.imgur.com/5D0R3mM.png)
![](https://i.imgur.com/HFkgbK4.png)
![](https://i.imgur.com/kdIdQEQ.png)
![](https://i.imgur.com/DVpSVlE.png)
![](https://i.imgur.com/Lsj5aN7.png)
![](https://i.imgur.com/KflQWmQ.png)
![](https://i.imgur.com/pXYSuAp.png)
![](https://i.imgur.com/JmJIRpE.png)
![](https://i.imgur.com/1Cfi4jx.png)
![](https://i.imgur.com/GXoksq0.png)
![](https://i.imgur.com/wxGGuZ9.png)
![](https://i.imgur.com/mt1VnDQ.png)
![](https://i.imgur.com/UVO7diU.png)
![](https://i.imgur.com/Zn99itX.png)
![](https://i.imgur.com/U1YRRv7.png)
![](https://i.imgur.com/IoTCqWQ.png)
![](https://i.imgur.com/LJgsS6D.png)
![](https://i.imgur.com/bNLHur3.png)
![](https://i.imgur.com/qX5bDqc.png)
![](https://i.imgur.com/evZTSK7.png)
![](https://i.imgur.com/vcFTHu7.png)
![](https://i.imgur.com/B0OxfFz.png)
## Genetative AI and Society
![](https://i.imgur.com/GZBqhzT.png)
![](https://i.imgur.com/cp5Jqld.png)
![](https://i.imgur.com/uQVir2J.png)
![](https://i.imgur.com/LoEnxmX.png)
![](https://i.imgur.com/kH5ueLs.png)
![](https://i.imgur.com/QJ8LZst.png)
![](https://i.imgur.com/lL6DjKd.png)
![](https://i.imgur.com/EkOSUOB.png)
![](https://i.imgur.com/i37ibIL.png)
![](https://i.imgur.com/8UUjY8A.png)
![](https://i.imgur.com/EFJcxZl.png)
![](https://i.imgur.com/puW6AYX.png)
- w3 meterials
- [he economic potential of generative AI: The next productivity frontier](https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier#introduction)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://arxiv.org/pdf/2303.10130.pdf)
- [The Potentially Large Effects of Artificial Intelligence on Economic Growth (Briggs/Kodnani)](https://www.gspublishing.com/content/research/en/reports/2023/03/27/d64e052b-0f6e-45d7-967b-d7be35fabd16.html)
- https://www.onetonline.org/
## Ref
- https://www.coursera.org/learn/generative-ai-for-everyone
- slides: https://community.deeplearning.ai/t/generative-ai-for-everyone-lecture-notes/481740
-

</body>
</html>


## What is Generative AI
![](https://i.imgur.com/Q877q0W.png)

![](https://i.imgur.com/8DpeAAb.png)

![](https://i.imgur.com/FhLUDBG.png)
- Supervised learning (labeling things)
![](https://i.imgur.com/fFv1p2L.png)
- 2010 - 2020: Large scale supervised learning
- LLM
	- How? supervised learning (A->B) 重複預測下一個word
![](https://i.imgur.com/d3JGgOK.png)
- 例子
	- 寫作：rewrite for clarity
	- 閱讀：有沒有在抱怨、情緒分析
	- 聊天：聊天機器人
- web search or LLM？
![](https://i.imgur.com/PIehfwc.png)

- LLM可能會錯，但回答比較精簡
	- web search有時會得到比較好的答案，但要花時間找到你要的資訊
## Generative AI Applications
![](https://i.imgur.com/ubs4shO.png)

![](https://i.imgur.com/ZblmQDs.png)
![](https://i.imgur.com/F1oNomh.png)
- setup
```python
import openai
import os

openai.api_key = os.getenv("OPENAI_API_KEY")

def llm_response(prompt):
    response = openai.ChatCompletion.create(
        model='gpt-3.5-turbo',
        messages=[{'role':'user','content':prompt}],
        temperature=0
    )
    return response.choices[0].message['content']
```
- classify
```python
prompt = '''
    Classify the following review 
    as having either a positive or
    negative sentiment:

    The banana pudding was really tasty!
'''

response = llm_response(prompt)
print(response)
```

```python
all_reviews = [
    'The mochi is excellent!',
    'Best soup dumplings I have ever eaten.',
    'Not worth the 3 month wait for a reservation.',
    'The colorful tablecloths made me smile!',
    'The pasta was cold.'
]

all_reviews
classifications = []
for review in all_reviews:
    prompt = f'''
        Classify the following review 
        as having either a positive or
        negative sentiment. State your answer
        as a single word, either "positive" or
        "negative":

        {review}
        '''
    response = llm_response(prompt)
    classifications.append(response)

classifications
```


![](https://i.imgur.com/MoG5DoG.png)

![](https://i.imgur.com/VOdC3dn.png)

![](https://i.imgur.com/zbDvhYJ.png)


![](https://i.imgur.com/ranyh2o.png)

![](https://i.imgur.com/BSS1Rsx.png)
### Advance technologies: Beyond prompting
![](https://i.imgur.com/EeOJZiM.png)

![](https://i.imgur.com/kkk2PLP.png)

![](https://i.imgur.com/Zx9Yun9.png)



![](https://i.imgur.com/EoqwkuW.png)

![](https://i.imgur.com/Ysjf6Bj.png)

![](https://i.imgur.com/cBEfLR0.png)

![](https://i.imgur.com/wdyy3ov.png)


![](https://i.imgur.com/4uomkXV.png)

![](https://i.imgur.com/VTx5KBM.png)

![](https://i.imgur.com/HM0jMCS.png)

![](https://i.imgur.com/DVqNRPm.png)

![](https://i.imgur.com/dnGp4io.png)

![](https://i.imgur.com/qOxdVE0.png)


![](https://i.imgur.com/JiWy6ky.png)

![](https://i.imgur.com/g9sp9gj.png)

![](https://i.imgur.com/DYyC9xa.png)

![](https://i.imgur.com/RmNpxgS.png)

![](https://i.imgur.com/YVs5iOT.png)

![](https://i.imgur.com/kvfmIsI.png)
![](https://i.imgur.com/s5j8JfV.png)

![](https://i.imgur.com/It3c1bn.png)

![](https://i.imgur.com/TrH6LIS.png)
## Generative AI and Business

![](https://i.imgur.com/5D0R3mM.png)
![](https://i.imgur.com/HFkgbK4.png)
![](https://i.imgur.com/kdIdQEQ.png)


![](https://i.imgur.com/DVpSVlE.png)

![](https://i.imgur.com/Lsj5aN7.png)


![](https://i.imgur.com/KflQWmQ.png)

![](https://i.imgur.com/pXYSuAp.png)

![](https://i.imgur.com/JmJIRpE.png)

![](https://i.imgur.com/1Cfi4jx.png)

![](https://i.imgur.com/GXoksq0.png)

![](https://i.imgur.com/wxGGuZ9.png)

![](https://i.imgur.com/mt1VnDQ.png)

![](https://i.imgur.com/UVO7diU.png)

![](https://i.imgur.com/Zn99itX.png)

![](https://i.imgur.com/U1YRRv7.png)

![](https://i.imgur.com/IoTCqWQ.png)

![](https://i.imgur.com/LJgsS6D.png)

![](https://i.imgur.com/bNLHur3.png)

![](https://i.imgur.com/qX5bDqc.png)

![](https://i.imgur.com/evZTSK7.png)

![](https://i.imgur.com/vcFTHu7.png)

![](https://i.imgur.com/B0OxfFz.png)

## Genetative AI and Society

![](https://i.imgur.com/GZBqhzT.png)

![](https://i.imgur.com/cp5Jqld.png)


![](https://i.imgur.com/uQVir2J.png)

![](https://i.imgur.com/LoEnxmX.png)
![](https://i.imgur.com/kH5ueLs.png)

![](https://i.imgur.com/QJ8LZst.png)

![](https://i.imgur.com/lL6DjKd.png)

![](https://i.imgur.com/EkOSUOB.png)


![](https://i.imgur.com/i37ibIL.png)

![](https://i.imgur.com/8UUjY8A.png)

![](https://i.imgur.com/EFJcxZl.png)

![](https://i.imgur.com/puW6AYX.png)

- w3 meterials
	- [he economic potential of generative AI: The next productivity frontier](https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier#introduction)
	- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://arxiv.org/pdf/2303.10130.pdf)
	- [The Potentially Large Effects of Artificial Intelligence on Economic Growth (Briggs/Kodnani)](https://www.gspublishing.com/content/research/en/reports/2023/03/27/d64e052b-0f6e-45d7-967b-d7be35fabd16.html)
	- https://www.onetonline.org/



## Ref
- https://www.coursera.org/learn/generative-ai-for-everyone
- slides: https://community.deeplearning.ai/t/generative-ai-for-everyone-lecture-notes/481740
- 

20231101-generative-ai-for-everyone

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

### 兩個很重要的領域 監督式學習及生成式AI
![](https://i.imgur.com/Wvb36Eh.png)
- 生成式AI 怎麼訓練？
- 使用很大的資料集用監督式學習方法訓練
![](https://i.imgur.com/8jvY67N.png)
### 過去監督式學習
![](https://i.imgur.com/Hs3EZ5U.png)
### 生成式AI會給workflow很大的改變
![](https://i.imgur.com/2kpX8iN.png)
### 未來三年的價值比較
![](https://i.imgur.com/qjjtLog.png)
## 機會
### 1. 長期價值？
##### Lensa 一個數位網紅的App 隨著時間熱度減退
##### 下一個Uber、Airbnb、Tinder 這種有“長期”價值的公司在哪？
![](https://i.imgur.com/hFrxWh8.png)
### 2. 客製化及不用程式碼的解決方案
##### 雖然廣告及搜尋價值最大，但也有一些專精特定領域的機會
e.g. 500億美元 pizza 專案、稻作生長等等
![](https://i.imgur.com/vqhohMr.png)
### 快速有效率建立一家公司 整合AI到現有業務
![](https://i.imgur.com/z4J9WEw.png)
### 建立startups的流程
![](https://i.imgur.com/Oy5Zv17.png)
### 想法很重要、不要直接到solution 且 構想要具體、可驗證
![](https://i.imgur.com/j7YzBtP.png)
### 負責任AI，不做有道德疑慮的方向
![](https://i.imgur.com/iykL9Dk.png)
## 風險
### AI 會有偏見 幻覺等問題、也會佔用到一些原本工作缺
![](https://i.imgur.com/SFL05ZA.png)
### AI 要達到人類智能可能還需要10年、且會慢慢發展不會一系之間突然變很厲害
![](https://i.imgur.com/asKyS80.png)
## Ref
- https://www.youtube.com/watch?v=5p248yoa3oE

</body>
</html>


### 兩個很重要的領域 監督式學習及生成式AI
![](https://i.imgur.com/Wvb36Eh.png)
- 生成式AI 怎麼訓練？
	- 使用很大的資料集用監督式學習方法訓練
![](https://i.imgur.com/8jvY67N.png)
### 過去監督式學習
![](https://i.imgur.com/Hs3EZ5U.png)
### 生成式AI會給workflow很大的改變
![](https://i.imgur.com/2kpX8iN.png)
### 未來三年的價值比較
![](https://i.imgur.com/qjjtLog.png)

## 機會
### 1. 長期價值？
##### Lensa 一個數位網紅的App 隨著時間熱度減退
##### 下一個Uber、Airbnb、Tinder 這種有“長期”價值的公司在哪？
![](https://i.imgur.com/hFrxWh8.png)
### 2. 客製化及不用程式碼的解決方案
##### 雖然廣告及搜尋價值最大，但也有一些專精特定領域的機會
e.g. 500億美元 pizza 專案、稻作生長等等
![](https://i.imgur.com/vqhohMr.png)
### 快速有效率建立一家公司 整合AI到現有業務
![](https://i.imgur.com/z4J9WEw.png)
### 建立startups的流程
![](https://i.imgur.com/Oy5Zv17.png)
### 想法很重要、不要直接到solution 且 構想要具體、可驗證
![](https://i.imgur.com/j7YzBtP.png)
### 負責任AI，不做有道德疑慮的方向
![](https://i.imgur.com/iykL9Dk.png)
## 風險
### AI 會有偏見 幻覺等問題、也會佔用到一些原本工作缺
![](https://i.imgur.com/SFL05ZA.png)
### AI 要達到人類智能可能還需要10年、且會慢慢發展不會一系之間突然變很厲害
![](https://i.imgur.com/asKyS80.png)

## Ref
- https://www.youtube.com/watch?v=5p248yoa3oE


Andrew-Ng-Opportunities-in-AI-2023

Building Systems with the ChatGPT API

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

有涵蓋LLM，由淺入深，有colab實作，還不錯
# Introduction
### Summarization
- [colab introduction](https://colab.research.google.com/drive/1cpdhABVmqmjLfJr-qQIlOOM0Yonc_xFY#scrollTo=Jd2i2959FP3L)
## Ref
- https://share.goodnotes.com/s/bXUxVzvrKQjpNVfT9gtLSt
- https://ai-guide.future.mozilla.org/
- colab
-

</body>
</html>


有涵蓋LLM，由淺入深，有colab實作，還不錯

# Introduction
### Summarization
	- [colab introduction](https://colab.research.google.com/drive/1cpdhABVmqmjLfJr-qQIlOOM0Yonc_xFY#scrollTo=Jd2i2959FP3L)

## Ref
- https://share.goodnotes.com/s/bXUxVzvrKQjpNVfT9gtLSt
- https://ai-guide.future.mozilla.org/
- colab

	- 


20231028-mozilla-ai

20231026-ChatGPT-Prompt-Engineering-for-Developers

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

小林說：分享有關於金融背景的ytr，分享得很棒 [我是如何快速学习一个领域的 (15:49)](https://www.youtube.com/watch?v=EViSmcCPK5g)
1. search 搜尋
1. 問題：資料太多
2. 整理框架 &#x3C; 結構化思維
3. 參考別人的框架
1. 看書目錄、框架
2. 可信來源 + 一文讀懂OOO
3. AI (chatgpt) 會錯 > 不斷交互問
4. 省略細節、注重重點
5. 製作影片有樹枝 > 塞細節
4. 一直問為什麼
1. 帶目的搜尋，忽略不重要的部分
5. 時間線
1. 起源
3. 太多資訊？ 找重點=> 結果導向數據 e.g. 大波動、市場真實反映=>股價
4. 一手訊息
6. 流程化思維
1. 大神有框架 有知識樹，問知識樹
7. 整理
1. 長文 讀兩次
1. 第一次忽略所有細節，找關鍵性框架訊息
2. 整理： 根據問題 想知道什麼細節 補充進去
8. 輸出
1. 複習 蓋起來重劃 這概念有點類似覆盤
![](https://i.imgur.com/3tdOHt7.png)
![](https://i.imgur.com/Ke28xnQ.png)
## Ref
- https://www.youtube.com/watch?v=EViSmcCPK5g
-

</body>
</html>


小林說：分享有關於金融背景的ytr，分享得很棒 [我是如何快速学习一个领域的 (15:49)](https://www.youtube.com/watch?v=EViSmcCPK5g)
1. search 搜尋
	1. 問題：資料太多
	2. 整理框架 < 結構化思維
	3. 參考別人的框架
		1. 看書目錄、框架
		2. 可信來源 + 一文讀懂OOO
		3. AI (chatgpt) 會錯 > 不斷交互問
		4. 省略細節、注重重點
		5. 製作影片有樹枝 > 塞細節
	4. 一直問為什麼
		1. 帶目的搜尋，忽略不重要的部分
	5. 時間線
		1. 起源
		3. 太多資訊？ 找重點=> 結果導向數據 e.g. 大波動、市場真實反映=>股價
		4. 一手訊息
	6. 流程化思維
		1. 大神有框架 有知識樹，問知識樹
	7. 整理
		1. 長文 讀兩次
			1. 第一次忽略所有細節，找關鍵性框架訊息
			2. 整理： 根據問題 想知道什麼細節 補充進去
	8. 輸出
		1. 複習 蓋起來重劃 這概念有點類似覆盤

![](https://i.imgur.com/3tdOHt7.png)
![](https://i.imgur.com/Ke28xnQ.png)

## Ref
- https://www.youtube.com/watch?v=EViSmcCPK5g
- 


20231023-learning

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

## TL;DR
LLM 可以幫助程式設計師包含：改進現有程式碼、簡化code、寫測試、優化程式除錯及解釋程式碼，使用流程為(1)建立prompt (2)取得結果 (3) 輸出結果，為了程式重複使用，會使用template機制讓呼叫比較簡單。
Palm - google 的llm模型
有三個組成
- API Key
- GEN AI library
model裡面選擇 chat bison vs text bison 差別是多次對話跟一次對話
大致上使用流程
1. create a prompt
2. completion
3. output result
## Getting start
```python
from utils import get_api_key
import os
import google.generativeai as palm
from google.api_core import client_options as client_options_lib
palm.configure(
api_key=get_api_key(),
transport="rest",
client_options=client_options_lib.ClientOptions(
api_endpoint=os.getenv("GOOGLE_API_BASE"),
)
)
models = [m for m in palm.list_models()
if 'generateText'
in m.supported_generation_methods]
model_bison = models[0]
model_bison
from google.api_core import retry
@retry.Retry()
def generate_text(prompt,
model=model_bison,
temperature=0.0):
return palm.generate_text(prompt=prompt,
model=model,
temperature=temperature)
```
### process
1. create a prompt `prompt = "Show me how to iterate across a list in Python."`
2. completion `completion = generate_text(prompt)`
3. output `print(completion.result)`
## String template
```python
import os
from utils import get_api_key
import google.generativeai as palm
from google.api_core import client_options as client_options_lib
palm.configure(
api_key=get_api_key(),
transport="rest",
client_options=client_options_lib.ClientOptions(
api_endpoint=os.getenv("GOOGLE_API_BASE"),
)
)
models = [m for m in palm.list_models() if 'generateText' in m.supported_generation_methods]
model_bison = models[0]
model_bison
from google.api_core import retry
@retry.Retry()
def generate_text(prompt,
model=model_bison,
temperature=0.0):
return palm.generate_text(prompt=prompt,
model=model,
temperature=temperature)
prompt_template = """
{priming}
{question}
{decorator}
Your solution:
"""
priming_text = "You are an expert at writing clear, concise, Python code."
question = "create a doubly linked list"
decorator = "Insert comments for each line of code."
prompt = prompt_template.format(priming=priming_text,
question=question,
decorator=decorator)
```
### prompt_template
- priming: 類似前情提要/背景
- question: 要問的問題
- decorator: 修飾回答樣式
## Pair Progaming scenarios
### 1. improve existing code
```python
prompt_template = """
I don't think this code is the best way to do it in Python, can you help me?
{question}
Please explain, in detail, what you did to improve it.
"""
question = """
def func_x(array)
for i in range(len(array)):
print(array[i])
"""
completion = generate_text(
prompt = prompt_template.format(question=question)
)
print(completion.result)
prompt_template = """
I don't think this code is the best way to do it in Python, can you help me?
{question}
Please explore ***multiple ways*** of solving the problem, and explain each.
"""
completion = generate_text(
prompt = prompt_template.format(question=question)
)
print(completion.result)
prompt_template = """
I don't think this code is the best way to do it in Python, can you help me?
{question}
Please explore multiple ways of solving the problem,
and tell me which is ***the most Pythonic***
"""
completion = generate_text(
prompt = prompt_template.format(question=question)
)
print(completion.result)
```
### 2. simplify code
```python
prompt_template = """
Can you please simplify this code for a linked list in Python?
{question}
Explain in detail what you did to modify it, and why.
"""
question = """
class Node:
def __init__(self, dataval=None):
self.dataval = dataval
self.nextval = None
class SLinkedList:
def __init__(self):
self.headval = None
list1 = SLinkedList()
list1.headval = Node("Mon")
e2 = Node("Tue")
e3 = Node("Wed")
list1.headval.nextval = e2
e2.nextval = e3
"""
completion = generate_text(
prompt = prompt_template.format(question=question)
)
print(completion.result)
```
### 3. write test cases
```python
prompt_template = """
Can you please create test cases in code for this Python code?
{question}
Explain in detail what these test cases are designed to achieve.
"""
# Note that the code I'm using here was output in the previous
# section. Your output code may be different.
question = """
class Node:
def __init__(self, dataval=None):
self.dataval = dataval
self.nextval = None
class SLinkedList:
def __init__(self):
self.head = None
def create_linked_list(data):
head = Node(data[0])
for i in range(1, len(data)):
node = Node(data[i])
node.nextval = head
head = node
return head
list1 = create_linked_list(["Mon", "Tue", "Wed"])
"""
completion = generate_text(
prompt = prompt_template.format(question=question)
)
print(completion.result)
```
### 4. Make code more efficient
```python
prompt_template = """
Can you please make this code more efficient?
{question}
Explain in detail what you changed and why.
"""
question = """
# Returns index of x in arr if present, else -1
def binary_search(arr, low, high, x):
# Check base case
if high >= low:
mid = (high + low) // 2
if arr[mid] == x:
return mid
elif arr[mid] > x:
return binary_search(arr, low, mid - 1, x)
else:
return binary_search(arr, mid + 1, high, x)
else:
return -1
# Test array
arr = [ 2, 3, 4, 10, 40 ]
x = 10
# Function call
result = binary_search(arr, 0, len(arr)-1, x)
if result != -1:
print("Element is present at index", str(result))
else:
print("Element is not present in array")
"""
completion = generate_text(
prompt = prompt_template.format(question=question)
)
print(completion.result)
```
有時候輸出有可能會錯
### 5. Debug your code
```python
prompt_template = """
Can you please help me to debug this code?
{question}
Explain in detail what you found and why it was a bug.
"""
# I deliberately introduced a bug into this code! Let's see if the LLM can find it.
# Note -- the model can't see this comment -- but the bug is in the
# print function. There's a circumstance where nodes can be null, and trying
# to print them would give a null error.
question = """
class Node:
def __init__(self, data):
self.data = data
self.next = None
self.prev = None
class doubly_linked_list:
def __init__(self):
self.head = None
# Adding data elements
def push(self, NewVal):
NewNode = Node(NewVal)
NewNode.next = self.head
if self.head is not None:
self.head.prev = NewNode
self.head = NewNode
# Print the Doubly Linked list in order
def listprint(self, node):
print(node.data),
last = node
node = node.next
dllist = doubly_linked_list()
dllist.push(12)
dllist.push(8)
dllist.push(62)
dllist.listprint(dllist.head)
"""
completion = generate_text(
prompt = prompt_template.format(question=question),
temperature = 0.7
)
print(completion.result)
```
- temperature>0會有一些隨機性，所以可能要檢查
###  Technical Debt
```python
import os
from utils import get_api_key
import google.generativeai as palm
from google.api_core import client_options as client_options_lib
palm.configure(
api_key=get_api_key(),
transport="rest",
client_options=client_options_lib.ClientOptions(
api_endpoint=os.getenv("GOOGLE_API_BASE"),
)
)
models = [m for m in palm.list_models() if 'generateText' in m.supported_generation_methods]
model_bison = models[0]
model_bison
from google.api_core import retry
@retry.Retry()
def generate_text(prompt,
model=model_bison,
temperature=0.0):
return palm.generate_text(prompt=prompt,
model=model,
temperature=temperature)
```
一個很複雜的code
```swift
#@title Complex Code Block
# Note: Taken from https://github.com/lmoroney/odmlbook/blob/63c0825094b2f44efc5c4d3226425a51990e73d6/BookSource/Chapter08/ios/cats_vs_dogs/CatVsDogClassifierSample/ModelDataHandler/ModelDataHandler.swift
CODE_BLOCK = """
// Copyright 2019 The TensorFlow Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
import CoreImage
import TensorFlowLite
import UIKit
/// An inference from invoking the `Interpreter`.
struct Inference {
let confidence: Float
let label: String
}
/// Information about a model file or labels file.
typealias FileInfo = (name: String, extension: String)
/// Information about the MobileNet model.
enum MobileNet {
static let modelInfo: FileInfo = (name: "converted_model", extension: "tflite")
}
/// This class handles all data preprocessing and makes calls to run inference on a given frame
/// by invoking the `Interpreter`. It then formats the inferences obtained and returns the top N
/// results for a successful inference.
class ModelDataHandler {
// MARK: - Public Properties
/// The current thread count used by the TensorFlow Lite Interpreter.
let threadCount: Int
let resultCount = 1
// MARK: - Model Parameters
let batchSize = 1
let inputChannels = 3
let inputWidth = 224
let inputHeight = 224
// MARK: - Private Properties
/// List of labels from the given labels file.
private var labels: [String] = ["Cat", "Dog"]
/// TensorFlow Lite `Interpreter` object for performing inference on a given model.
private var interpreter: Interpreter
/// Information about the alpha component in RGBA data.
private let alphaComponent = (baseOffset: 4, moduloRemainder: 3)
// MARK: - Initialization
/// A failable initializer for `ModelDataHandler`. A new instance is created if the model and
/// labels files are successfully loaded from the app's main bundle. Default `threadCount` is 1.
init?(modelFileInfo: FileInfo, threadCount: Int = 1) {
let modelFilename = modelFileInfo.name
// Construct the path to the model file.
guard let modelPath = Bundle.main.path(
forResource: modelFilename,
ofType: modelFileInfo.extension
) else {
print("Failed to load the model file with name: \(modelFilename).")
return nil
}
// Specify the options for the `Interpreter`.
self.threadCount = threadCount
var options = InterpreterOptions()
options.threadCount = threadCount
do {
// Create the `Interpreter`.
interpreter = try Interpreter(modelPath: modelPath, options: options)
} catch let error {
print("Failed to create the interpreter with error: \(error.localizedDescription)")
return nil
}
}
// MARK: - Public Methods
/// Performs image preprocessing, invokes the `Interpreter`, and process the inference results.
func runModel(onFrame pixelBuffer: CVPixelBuffer) -> [Inference]? {
let sourcePixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer)
assert(sourcePixelFormat == kCVPixelFormatType_32ARGB ||
sourcePixelFormat == kCVPixelFormatType_32BGRA ||
sourcePixelFormat == kCVPixelFormatType_32RGBA)
let imageChannels = 4
assert(imageChannels >= inputChannels)
// Crops the image to the biggest square in the center and scales it down to model dimensions.
let scaledSize = CGSize(width: inputWidth, height: inputHeight)
guard let thumbnailPixelBuffer = pixelBuffer.centerThumbnail(ofSize: scaledSize) else {
return nil
}
let outputTensor: Tensor
do {
// Allocate memory for the model's input `Tensor`s.
try interpreter.allocateTensors()
// Remove the alpha component from the image buffer to get the RGB data.
guard let rgbData = rgbDataFromBuffer(
thumbnailPixelBuffer,
byteCount: batchSize * inputWidth * inputHeight * inputChannels
) else {
print("Failed to convert the image buffer to RGB data.")
return nil
}
// Copy the RGB data to the input `Tensor`.
try interpreter.copy(rgbData, toInputAt: 0)
// Run inference by invoking the `Interpreter`.
try interpreter.invoke()
// Get the output `Tensor` to process the inference results.
outputTensor = try interpreter.output(at: 0)
} catch let error {
print("Failed to invoke the interpreter with error: \(error.localizedDescription)")
return nil
}
let results = [Float32](unsafeData: outputTensor.data) ?? []
// Process the results.
let topNInferences = getTopN(results: results)
// Return the inference time and inference results.
return topNInferences
}
// MARK: - Private Methods
/// Returns the top N inference results sorted in descending order.
private func getTopN(results: [Float]) -> [Inference] {
// Create a zipped array of tuples [(labelIndex: Int, confidence: Float)].
let zippedResults = zip(labels.indices, results)
// Sort the zipped results by confidence value in descending order.
let sortedResults = zippedResults.sorted { $0.1 > $1.1 }.prefix(resultCount)
// Return the `Inference` results.
return sortedResults.map { result in Inference(confidence: result.1, label: labels[result.0]) }
}
/// Loads the labels from the labels file and stores them in the `labels` property.
private func loadLabels(fileInfo: FileInfo) {
let filename = fileInfo.name
let fileExtension = fileInfo.extension
guard let fileURL = Bundle.main.url(forResource: filename, withExtension: fileExtension) else {
fatalError("Labels file not found in bundle. Please add a labels file with name " +
"\(filename).\(fileExtension) and try again.")
}
do {
let contents = try String(contentsOf: fileURL, encoding: .utf8)
labels = contents.components(separatedBy: .newlines)
} catch {
fatalError("Labels file named \(filename).\(fileExtension) cannot be read. Please add a " +
"valid labels file and try again.")
}
}
/// Returns the RGB data representation of the given image buffer with the specified `byteCount`.
///
/// - Parameters
/// - buffer: The pixel buffer to convert to RGB data.
/// - byteCount: The expected byte count for the RGB data calculated using the values that the
/// model was trained on: `batchSize * imageWidth * imageHeight * componentsCount`.
/// - isModelQuantized: Whether the model is quantized (i.e. fixed point values rather than
/// floating point values).
/// - Returns: The RGB data representation of the image buffer or `nil` if the buffer could not be
/// converted.
private func rgbDataFromBuffer(
_ buffer: CVPixelBuffer,
byteCount: Int
) -> Data? {
CVPixelBufferLockBaseAddress(buffer, .readOnly)
defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) }
guard let mutableRawPointer = CVPixelBufferGetBaseAddress(buffer) else {
return nil
}
let count = CVPixelBufferGetDataSize(buffer)
let bufferData = Data(bytesNoCopy: mutableRawPointer, count: count, deallocator: .none)
var rgbBytes = [Float](repeating: 0, count: byteCount)
var index = 0
for component in bufferData.enumerated() {
let offset = component.offset
let isAlphaComponent = (offset % alphaComponent.baseOffset) == alphaComponent.moduloRemainder
guard !isAlphaComponent else { continue }
rgbBytes[index] = Float(component.element) / 255.0
index += 1
}
return rgbBytes.withUnsafeBufferPointer(Data.init)
}
}
// MARK: - Extensions
extension Data {
/// Creates a new buffer by copying the buffer pointer of the given array.
///
/// - Warning: The given array's element type `T` must be trivial in that it can be copied bit
/// for bit with no indirection or reference-counting operations; otherwise, reinterpreting
/// data from the resulting buffer has undefined behavior.
/// - Parameter array: An array with elements of type `T`.
init
<div>
  (copyingBufferOf array: [T]) {
  self = array.withUnsafeBufferPointer(Data.init)
  }
  }
  extension Array {
  /// Creates a new array from the bytes of the given unsafe data.
  ///
  /// - Warning: The array's `Element` type must be trivial in that it can be copied bit for bit
  /// with no indirection or reference-counting operations; otherwise, copying the raw bytes in
  /// the `unsafeData`'s buffer to a new array returns an unsafe copy.
  /// - Note: Returns `nil` if `unsafeData.count` is not a multiple of
  /// `MemoryLayout
  <div>
    .stride`.
    /// - Parameter unsafeData: The data containing the bytes to turn into an array.
    init?(unsafeData: Data) {
    guard unsafeData.count % MemoryLayout
    <div>
      .stride == 0 else { return nil }
      #if swift(>=5.0)
      self = unsafeData.withUnsafeBytes { .init($0.bindMemory(to: Element.self)) }
      #else
      self = unsafeData.withUnsafeBytes {
      .init(UnsafeBufferPointer
      <div>
        (
        start: $0,
        count: unsafeData.count / MemoryLayout
        <div>
          .stride
          ))
          }
          #endif // swift(>=5.0)
          }
          }
          """
          ```
          ```python
          prompt_template = """
          Can you please explain how this code works?
          {question}
          Use a lot of detail and make it as clear as possible.
          """
          completion = generate_text(
          prompt = prompt_template.format(question=CODE_BLOCK)
          )
          print(completion.result)
          ```
          ## Ref
          - https://learn.deeplearning.ai/pair-programming-llm/
        </div>
      </div>
    </div>
  </div>
</div>

</body>
</html>


## TL;DR
LLM 可以幫助程式設計師包含：改進現有程式碼、簡化code、寫測試、優化程式除錯及解釋程式碼，使用流程為(1)建立prompt (2)取得結果 (3) 輸出結果，為了程式重複使用，會使用template機制讓呼叫比較簡單。

Palm - google 的llm模型
有三個組成
- API Key
- GEN AI library
model裡面選擇 chat bison vs text bison 差別是多次對話跟一次對話
大致上使用流程
1. create a prompt
2. completion
3. output result
## Getting start
```python
from utils import get_api_key
import os
import google.generativeai as palm
from google.api_core import client_options as client_options_lib

palm.configure(
    api_key=get_api_key(),
    transport="rest",
    client_options=client_options_lib.ClientOptions(
        api_endpoint=os.getenv("GOOGLE_API_BASE"),
    )
)
models = [m for m in palm.list_models() 
          if 'generateText' 
          in m.supported_generation_methods]
model_bison = models[0]
model_bison
from google.api_core import retry
@retry.Retry()
def generate_text(prompt,
                  model=model_bison,
                  temperature=0.0):
    return palm.generate_text(prompt=prompt,
                              model=model,
                              temperature=temperature)
```

### process
1. create a prompt `prompt = "Show me how to iterate across a list in Python."`
2. completion `completion = generate_text(prompt)`
3. output `print(completion.result)`
## String template
```python
import os
from utils import get_api_key
import google.generativeai as palm
from google.api_core import client_options as client_options_lib

palm.configure(
    api_key=get_api_key(),
    transport="rest",
    client_options=client_options_lib.ClientOptions(
        api_endpoint=os.getenv("GOOGLE_API_BASE"),
    )
)
models = [m for m in palm.list_models() if 'generateText' in m.supported_generation_methods]
model_bison = models[0]
model_bison
from google.api_core import retry
@retry.Retry()
def generate_text(prompt, 
                  model=model_bison, 
                  temperature=0.0):
    return palm.generate_text(prompt=prompt,
                              model=model,
                              temperature=temperature)
prompt_template = """
{priming}

{question}

{decorator}

Your solution:
"""
priming_text = "You are an expert at writing clear, concise, Python code."
question = "create a doubly linked list"
decorator = "Insert comments for each line of code."
prompt = prompt_template.format(priming=priming_text,
                                question=question,
                                decorator=decorator)
```
### prompt_template
- priming: 類似前情提要/背景
- question: 要問的問題
- decorator: 修飾回答樣式
## Pair Progaming scenarios
### 1. improve existing code
```python
prompt_template = """
I don't think this code is the best way to do it in Python, can you help me?

{question}

Please explain, in detail, what you did to improve it.
"""
question = """
def func_x(array)
  for i in range(len(array)):
    print(array[i])
"""
completion = generate_text(
    prompt = prompt_template.format(question=question)
)
print(completion.result)
prompt_template = """
I don't think this code is the best way to do it in Python, can you help me?

{question}

Please explore ***multiple ways*** of solving the problem, and explain each.
"""
completion = generate_text(
    prompt = prompt_template.format(question=question)
)
print(completion.result)
prompt_template = """
I don't think this code is the best way to do it in Python, can you help me?

{question}

Please explore multiple ways of solving the problem, 
and tell me which is ***the most Pythonic***
"""
completion = generate_text(
    prompt = prompt_template.format(question=question)
)
print(completion.result)
```
### 2. simplify code
```python
prompt_template = """
Can you please simplify this code for a linked list in Python?

{question}

Explain in detail what you did to modify it, and why.
"""
question = """
class Node:
  def __init__(self, dataval=None):
    self.dataval = dataval
    self.nextval = None

class SLinkedList:
  def __init__(self):
    self.headval = None

list1 = SLinkedList()
list1.headval = Node("Mon")
e2 = Node("Tue")
e3 = Node("Wed")
list1.headval.nextval = e2
e2.nextval = e3

"""
completion = generate_text(
    prompt = prompt_template.format(question=question)
)
print(completion.result)
```
### 3. write test cases
```python
prompt_template = """
Can you please create test cases in code for this Python code?

{question}

Explain in detail what these test cases are designed to achieve.
"""
# Note that the code I'm using here was output in the previous
# section. Your output code may be different.
question = """
class Node:
  def __init__(self, dataval=None):
    self.dataval = dataval
    self.nextval = None

class SLinkedList:
  def __init__(self):
    self.head = None

def create_linked_list(data):
  head = Node(data[0])
  for i in range(1, len(data)):
    node = Node(data[i])
    node.nextval = head
    head = node
  return head

list1 = create_linked_list(["Mon", "Tue", "Wed"])
"""
completion = generate_text(
    prompt = prompt_template.format(question=question)
)
print(completion.result)
```
### 4. Make code more efficient
```python
prompt_template = """
Can you please make this code more efficient?

{question}

Explain in detail what you changed and why.
"""
question = """
# Returns index of x in arr if present, else -1
def binary_search(arr, low, high, x):
    # Check base case
    if high >= low:
        mid = (high + low) // 2
        if arr[mid] == x:
            return mid
        elif arr[mid] > x:
            return binary_search(arr, low, mid - 1, x)
        else:
            return binary_search(arr, mid + 1, high, x)
    else:
        return -1

# Test array
arr = [ 2, 3, 4, 10, 40 ]
x = 10

# Function call
result = binary_search(arr, 0, len(arr)-1, x)

if result != -1:
    print("Element is present at index", str(result))
else:
    print("Element is not present in array")

"""
completion = generate_text(
    prompt = prompt_template.format(question=question)
)
print(completion.result)
```
有時候輸出有可能會錯
### 5.  Debug your code
```python
prompt_template = """
Can you please help me to debug this code?

{question}

Explain in detail what you found and why it was a bug.
"""
# I deliberately introduced a bug into this code! Let's see if the LLM can find it.
# Note -- the model can't see this comment -- but the bug is in the
# print function. There's a circumstance where nodes can be null, and trying
# to print them would give a null error.
question = """
class Node:
   def __init__(self, data):
      self.data = data
      self.next = None
      self.prev = None

class doubly_linked_list:
   def __init__(self):
      self.head = None

# Adding data elements
   def push(self, NewVal):
      NewNode = Node(NewVal)
      NewNode.next = self.head
      if self.head is not None:
         self.head.prev = NewNode
      self.head = NewNode

# Print the Doubly Linked list in order
   def listprint(self, node):
       print(node.data),
       last = node
       node = node.next

dllist = doubly_linked_list()
dllist.push(12)
dllist.push(8)
dllist.push(62)
dllist.listprint(dllist.head)
"""
completion = generate_text(
    prompt = prompt_template.format(question=question),
    temperature = 0.7
)
print(completion.result)
```
- temperature>0會有一些隨機性，所以可能要檢查
###  Technical Debt

```python
import os
from utils import get_api_key
import google.generativeai as palm
from google.api_core import client_options as client_options_lib

palm.configure(
    api_key=get_api_key(),
    transport="rest",
    client_options=client_options_lib.ClientOptions(
        api_endpoint=os.getenv("GOOGLE_API_BASE"),
    )
)
models = [m for m in palm.list_models() if 'generateText' in m.supported_generation_methods]
model_bison = models[0]
model_bison
from google.api_core import retry
@retry.Retry()
def generate_text(prompt, 
                  model=model_bison, 
                  temperature=0.0):
    return palm.generate_text(prompt=prompt,
                              model=model,
                              temperature=temperature)
```
一個很複雜的code
```swift
#@title Complex Code Block
# Note: Taken from https://github.com/lmoroney/odmlbook/blob/63c0825094b2f44efc5c4d3226425a51990e73d6/BookSource/Chapter08/ios/cats_vs_dogs/CatVsDogClassifierSample/ModelDataHandler/ModelDataHandler.swift
CODE_BLOCK = """
// Copyright 2019 The TensorFlow Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//    http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import CoreImage
import TensorFlowLite
import UIKit


/// An inference from invoking the `Interpreter`.
struct Inference {
  let confidence: Float
  let label: String
}

/// Information about a model file or labels file.
typealias FileInfo = (name: String, extension: String)

/// Information about the MobileNet model.
enum MobileNet {
  static let modelInfo: FileInfo = (name: "converted_model", extension: "tflite")
}

/// This class handles all data preprocessing and makes calls to run inference on a given frame
/// by invoking the `Interpreter`. It then formats the inferences obtained and returns the top N
/// results for a successful inference.
class ModelDataHandler {

  // MARK: - Public Properties

  /// The current thread count used by the TensorFlow Lite Interpreter.
  let threadCount: Int

  let resultCount = 1

  // MARK: - Model Parameters

  let batchSize = 1
  let inputChannels = 3
  let inputWidth = 224
  let inputHeight = 224

  // MARK: - Private Properties

  /// List of labels from the given labels file.
  private var labels: [String] = ["Cat", "Dog"]

  /// TensorFlow Lite `Interpreter` object for performing inference on a given model.
  private var interpreter: Interpreter

  /// Information about the alpha component in RGBA data.
  private let alphaComponent = (baseOffset: 4, moduloRemainder: 3)

  // MARK: - Initialization

  /// A failable initializer for `ModelDataHandler`. A new instance is created if the model and
  /// labels files are successfully loaded from the app's main bundle. Default `threadCount` is 1.
  init?(modelFileInfo: FileInfo, threadCount: Int = 1) {
    let modelFilename = modelFileInfo.name

    // Construct the path to the model file.
    guard let modelPath = Bundle.main.path(
      forResource: modelFilename,
      ofType: modelFileInfo.extension
      ) else {
        print("Failed to load the model file with name: \(modelFilename).")
        return nil
    }

    // Specify the options for the `Interpreter`.
    self.threadCount = threadCount
    var options = InterpreterOptions()
    options.threadCount = threadCount
    do {
      // Create the `Interpreter`.
      interpreter = try Interpreter(modelPath: modelPath, options: options)
    } catch let error {
      print("Failed to create the interpreter with error: \(error.localizedDescription)")
      return nil
    }

  }

  // MARK: - Public Methods

  /// Performs image preprocessing, invokes the `Interpreter`, and process the inference results.
  func runModel(onFrame pixelBuffer: CVPixelBuffer) -> [Inference]? {
    let sourcePixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer)
    assert(sourcePixelFormat == kCVPixelFormatType_32ARGB ||
      sourcePixelFormat == kCVPixelFormatType_32BGRA ||
      sourcePixelFormat == kCVPixelFormatType_32RGBA)


    let imageChannels = 4
    assert(imageChannels >= inputChannels)

    // Crops the image to the biggest square in the center and scales it down to model dimensions.
    let scaledSize = CGSize(width: inputWidth, height: inputHeight)
    guard let thumbnailPixelBuffer = pixelBuffer.centerThumbnail(ofSize: scaledSize) else {
      return nil
    }

    let outputTensor: Tensor
    do {
      // Allocate memory for the model's input `Tensor`s.
      try interpreter.allocateTensors()

      // Remove the alpha component from the image buffer to get the RGB data.
      guard let rgbData = rgbDataFromBuffer(
        thumbnailPixelBuffer,
        byteCount: batchSize * inputWidth * inputHeight * inputChannels
        ) else {
          print("Failed to convert the image buffer to RGB data.")
          return nil
      }

      // Copy the RGB data to the input `Tensor`.
      try interpreter.copy(rgbData, toInputAt: 0)

      // Run inference by invoking the `Interpreter`.
      try interpreter.invoke()

      // Get the output `Tensor` to process the inference results.
      outputTensor = try interpreter.output(at: 0)
    } catch let error {
      print("Failed to invoke the interpreter with error: \(error.localizedDescription)")
      return nil
    }

    let results = [Float32](unsafeData: outputTensor.data) ?? []

    // Process the results.
    let topNInferences = getTopN(results: results)

    // Return the inference time and inference results.
    return topNInferences
  }

  // MARK: - Private Methods

  /// Returns the top N inference results sorted in descending order.
  private func getTopN(results: [Float]) -> [Inference] {
    // Create a zipped array of tuples [(labelIndex: Int, confidence: Float)].
    let zippedResults = zip(labels.indices, results)

    // Sort the zipped results by confidence value in descending order.
    let sortedResults = zippedResults.sorted { $0.1 > $1.1 }.prefix(resultCount)

    // Return the `Inference` results.
    return sortedResults.map { result in Inference(confidence: result.1, label: labels[result.0]) }
  }

  /// Loads the labels from the labels file and stores them in the `labels` property.
  private func loadLabels(fileInfo: FileInfo) {
    let filename = fileInfo.name
    let fileExtension = fileInfo.extension
    guard let fileURL = Bundle.main.url(forResource: filename, withExtension: fileExtension) else {
      fatalError("Labels file not found in bundle. Please add a labels file with name " +
        "\(filename).\(fileExtension) and try again.")
    }
    do {
      let contents = try String(contentsOf: fileURL, encoding: .utf8)
      labels = contents.components(separatedBy: .newlines)
    } catch {
      fatalError("Labels file named \(filename).\(fileExtension) cannot be read. Please add a " +
        "valid labels file and try again.")
    }
  }

  /// Returns the RGB data representation of the given image buffer with the specified `byteCount`.
  ///
  /// - Parameters
  ///   - buffer: The pixel buffer to convert to RGB data.
  ///   - byteCount: The expected byte count for the RGB data calculated using the values that the
  ///       model was trained on: `batchSize * imageWidth * imageHeight * componentsCount`.
  ///   - isModelQuantized: Whether the model is quantized (i.e. fixed point values rather than
  ///       floating point values).
  /// - Returns: The RGB data representation of the image buffer or `nil` if the buffer could not be
  ///     converted.
  private func rgbDataFromBuffer(
    _ buffer: CVPixelBuffer,
    byteCount: Int
    ) -> Data? {
    CVPixelBufferLockBaseAddress(buffer, .readOnly)
    defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) }
    guard let mutableRawPointer = CVPixelBufferGetBaseAddress(buffer) else {
      return nil
    }
    let count = CVPixelBufferGetDataSize(buffer)
    let bufferData = Data(bytesNoCopy: mutableRawPointer, count: count, deallocator: .none)
    var rgbBytes = [Float](repeating: 0, count: byteCount)
    var index = 0
    for component in bufferData.enumerated() {
      let offset = component.offset
      let isAlphaComponent = (offset % alphaComponent.baseOffset) == alphaComponent.moduloRemainder
      guard !isAlphaComponent else { continue }
      rgbBytes[index] = Float(component.element) / 255.0
      index += 1
    }

    return rgbBytes.withUnsafeBufferPointer(Data.init)

  }
}

// MARK: - Extensions

extension Data {
  /// Creates a new buffer by copying the buffer pointer of the given array.
  ///
  /// - Warning: The given array's element type `T` must be trivial in that it can be copied bit
  ///     for bit with no indirection or reference-counting operations; otherwise, reinterpreting
  ///     data from the resulting buffer has undefined behavior.
  /// - Parameter array: An array with elements of type `T`.
  init<T>(copyingBufferOf array: [T]) {
    self = array.withUnsafeBufferPointer(Data.init)
  }
}

extension Array {
  /// Creates a new array from the bytes of the given unsafe data.
  ///
  /// - Warning: The array's `Element` type must be trivial in that it can be copied bit for bit
  ///     with no indirection or reference-counting operations; otherwise, copying the raw bytes in
  ///     the `unsafeData`'s buffer to a new array returns an unsafe copy.
  /// - Note: Returns `nil` if `unsafeData.count` is not a multiple of
  ///     `MemoryLayout<Element>.stride`.
  /// - Parameter unsafeData: The data containing the bytes to turn into an array.
  init?(unsafeData: Data) {

    guard unsafeData.count % MemoryLayout<Element>.stride == 0 else { return nil }
    #if swift(>=5.0)
    self = unsafeData.withUnsafeBytes { .init($0.bindMemory(to: Element.self)) }
    #else
    self = unsafeData.withUnsafeBytes {
      .init(UnsafeBufferPointer<Element>(
        start: $0,
        count: unsafeData.count / MemoryLayout<Element>.stride
      ))
    }
    #endif  // swift(>=5.0)
  }
}
"""
```
```python
prompt_template = """
Can you please explain how this code works?

{question}

Use a lot of detail and make it as clear as possible.
"""
completion = generate_text(
    prompt = prompt_template.format(question=CODE_BLOCK)
)
print(completion.result)
```

## Ref
- https://learn.deeplearning.ai/pair-programming-llm/

20231023-Pair-Programming-with-a-Large-Language-Model

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Contents</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

web 生物辨識api
## Ref

</body>
</html>
