# This should be placed before the FastAPI app is initialized.
llm = LLM(model=os.path.expanduser('./'), load_format="safetensors", enforce_eager=True)

app = FastAPI()

@app.get("/")
async def root():
    return "OK"

@app.post("/infer")
async def infer(request: Request):
    # Inference code
    request_data = await request.json()
    prompts = request_data["prompt"]
    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
    outputs = llm.generate(prompts, sampling_params)
    return {
        "response" : outputs[^0].outputs[^0].text
    }
```


#### **Streaming Responses with Server-Sent Events (SSE)**

To stream responses, you must first update your `dat1.yaml` to set the response type to `sse`.

```yaml
model_name: chat_completion
response_type: sse
exclude:
  - '**/.git/**'
  - '**/.idea/**'
```

Next, modify your `handler.py` to use `EventSourceResponse` to return a generator that streams data.

```python
from fastapi import Request, FastAPI
from sse_starlette.sse import EventSourceResponse
import json

app = FastAPI()

@app.get("/")
async def root():
    return "OK"

async def response_generator():
    for i in range(10):
        yield json.dumps({"response": f"Response {i}"})

@app.post("/infer")
async def infer(request: Request):
    return EventSourceResponse(response_generator(), sep="\n")
```


## Platform Documentation

### Pricing

Dat1 utilizes a pay-per-second billing model. Customers are only charged for the seconds their model is actively processing requests (inference time). There are no charges for idle time, model loading time, or timeouts. This model is designed to be cost-effective for applications with variable or intermittent traffic.

### Privacy Policy

The platform is fully compliant with GDPR and CCPA. Dat1 adheres to top cybersecurity best practices to ensure the privacy and security of user data and models. The privacy policy details data handling procedures, user rights, and compliance measures.

## Optional Resources

### Reddit AMA Summary

In a discussion with the Reddit community, one of the founders provided insights into the platform's technical details and future roadmap. Key topics included the architecture behind the low cold-start times, plans for supporting new model types, and the company's vision for making large-scale AI more accessible.

### EU-Startups Profile

Dat1 is featured on EU-Startups as a promising company in the AI infrastructure space. The profile provides a business summary and an elevator pitch, highlighting its value proposition of providing serverless GPU infrastructure for generative AI companies.