graphdoc.prompts package
- class graphdoc.prompts.BadDocGeneratorSignature(*, database_schema: str, documented_schema: str)[source]
Bases:
Signature### TASK: Given a GraphQL Schema, generate intentionally incorrect documentation for the columns of the tables in the database.
### Requirements: - Every table, entity, enum, etc. must have at least one column with a description that is obviosly incorrect. - The documentation must be incorrect and misleading. - The documentation should be scattered, with only some columns having documentation.
### Formatting - Ensure that the schema maintains proper documentation formatting, as is provided.
- database_schema: str
- documented_schema: str
- model_config: ClassVar[ConfigDict] = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class graphdoc.prompts.DocGeneratorHelperSignature(*, database_schema: str, documented_schema: str)[source]
Bases:
Signature### TASK: Analyze the provided GraphQL Schema and generate detailed yet concise descriptions for each field within the database tables and enums.
### Requirements: - If the field is unclear, and the documentation result is ambiguous, request additional information: “WARNING: Please provide additional information to avoid confusion”. - Utilize only the verified information from the schema to ensure accuracy. - Descriptions should be factual, straightforward, and avoid any speculative language. - Refrain from using the phrase “in the { table } table” within your descriptions. - Ensure that the documentation adheres to standard schema formatting without modifying the underlying schema structure.
### Formatting: - Maintain consistency with the existing documentation style and structure. - Focus on clarity and precision to aid developers and system architects in understanding the schema’s components effectively.
- database_schema: str
- documented_schema: str
- model_config: ClassVar[ConfigDict] = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class graphdoc.prompts.DocGeneratorPrompt(prompt: str | Signature | SignatureMeta, prompt_type: Literal['predict', 'chain_of_thought'] | Callable, prompt_metric: DocQualityPrompt)[source]
Bases:
SinglePromptDocGeneratorPrompt class for generating documentation for GraphQL schemas.
- evaluate_documentation_quality(schema: Example, pred: Prediction, trace=None, scalar=True) int[source]
Evaluate the quality of the documentation. Utilizes the instantiated metric type to evaluate the quality of the documentation.
- Parameters:
schema (dspy.Example) – The schema to evaluate the documentation for.
pred (dspy.Prediction) – The predicted documentation.
trace (Any) – The trace of the prediction.
scalar (bool) – Whether to return a squared score or the full evaluation object.
- Returns:
The squared score or the full evaluation object.
- Return type:
- evaluate_metric(example: Example, prediction: Prediction, trace=None) Any[source]
This is the metric used to evalaute the prompt.
- Parameters:
example (dspy.Example) – The example to evaluate the metric on.
prediction (dspy.Prediction) – The prediction to evaluate the metric on.
trace (Any) – The trace to evaluate the metric on. This is for DSPy.
- format_metric(examples: List[Example], overall_score: float, results: List, scores: List) Dict[str, Any][source]
Format the metric results into a dictionary.
- Parameters:
examples (List[dspy.Example]) – The examples used to evaluate the metric.
overall_score (float) – The overall score of the metric.
results (List) – The results of the metric.
scores (List) – The scores of the metric.
- class graphdoc.prompts.DocGeneratorSignature(*, database_schema: str, documented_schema: str)[source]
Bases:
Signature### TASK: Analyze the provided GraphQL Schema and generate detailed yet concise descriptions for each field within the database tables and enums.
### Requirements: - Utilize only the verified information from the schema to ensure accuracy. - Descriptions should be factual, straightforward, and avoid any speculative language. - Refrain from using the phrase “in the { table } table” within your descriptions. - Ensure that the documentation adheres to standard schema formatting without modifying the underlying schema structure. - Make sure that the entities themselves are documented.
### Formatting: - Maintain consistency with the existing documentation style and structure. - Focus on clarity and precision to aid developers and system architects in understanding the schema’s components effectively.
- database_schema: str
- documented_schema: str
- model_config: ClassVar[ConfigDict] = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class graphdoc.prompts.DocQualityDemonstrationSignature(*, database_schema: str, category: Literal['perfect', 'almost perfect', 'poor but correct', 'incorrect'], rating: Literal[4, 3, 2, 1])[source]
Bases:
SignatureYou are evaluating the output of an LLM program, expect hallucinations. Given a GraphQL Schema, evaluate the quality of documentation for that schema and provide a category rating.
The categories are described as: - perfect (4): The documentation contains enough information so that the interpretation of the schema and its database content is completely free of ambiguity.
perfect (4) example: type Domain @entity {
“ The namehash (id) of the parent name. References the Domain entity that is the parent of the current domain. Type: Domain ” parent: Domain
}
- almost perfect (3): The documentation is almost perfect and free from ambiguity, but there is room for improvement.
almost perfect (3) example: type Token @entity {
“ Name of the token, mirrored from the smart contract ” name: String!
}
- poor but correct (2): The documentation is poor but correct and has room for improvement due to missing information. The documentation is not incorrect.
poor but correct (2) example: type InterestRate @entity {
“Description for column: id” id: ID!
}
- incorrect (1): The documentation is incorrect and contains inaccurate or misleading information. Any incorrect information automatically leads to an incorrect rating, even if some correct information is present.
incorrect (1) example: type BridgeProtocol implements Protocol @entity {
“ Social Security Number of the protocol’s main developer ” id: Bytes!
}
Output a number rating that corresponds to the categories described above.
- database_schema: str
- category: Literal['perfect', 'almost perfect', 'poor but correct', 'incorrect']
- rating: Literal[4, 3, 2, 1]
- model_config: ClassVar[ConfigDict] = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class graphdoc.prompts.DocQualityPrompt(prompt: Literal['doc_quality', 'doc_quality_demo'] | Signature | SignatureMeta = 'doc_quality', prompt_type: Literal['predict', 'chain_of_thought'] | Callable = 'predict', prompt_metric: Literal['rating', 'category'] | Callable = 'rating')[source]
Bases:
SinglePromptDocQualityPrompt class for evaluating documentation quality.
This is a single prompt that can be used to evaluate the quality of the documentation for a given schema. This is a wrapper around the SinglePrompt class that implements the abstract methods.
- __init__(prompt: Literal['doc_quality', 'doc_quality_demo'] | Signature | SignatureMeta = 'doc_quality', prompt_type: Literal['predict', 'chain_of_thought'] | Callable = 'predict', prompt_metric: Literal['rating', 'category'] | Callable = 'rating') None[source]
Initialize the DocQualityPrompt.
- Parameters:
prompt (Union[str, dspy.Signature]) – The prompt to use. Can either be a string that maps to a defined signature, as set in the doc_quality_factory, or a dspy.Signature.
prompt_type (Union[Literal["predict", "chain_of_thought"], Callable]) – The type of prompt to use.
prompt_metric (Union[Literal["rating", "category"], Callable]) – The metric to use. Can either be a string that maps to a defined metric, as set in the doc_quality_factory, or a custom callable function. Function must have the signature (example: dspy.Example, prediction: dspy.Prediction) -> bool.
- evaluate_metric(example: Example, prediction: Prediction, trace=None) bool[source]
Evaluate the metric for the given example and prediction.
- Parameters:
example (dspy.Example) – The example to evaluate the metric on.
prediction (dspy.Prediction) – The prediction to evaluate the metric on.
trace (Any) – Used for DSPy.
- Returns:
The result of the evaluation. A boolean for if the metric is correct.
- Return type:
- format_metric(examples: List[Example], overall_score: float, results: List, scores: List) Dict[str, Any][source]
Formats evaluation metrics into a structured report containing: - Overall score across all categories - Percentage correct per category - Detailed results for each evaluation
- Parameters:
examples (List[dspy.Example]) – The examples to evaluate the metric on.
overall_score (float) – The overall score across all categories.
results (List) – The results of the evaluation.
scores (List) – The scores of the evaluation.
- Returns:
A dictionary containing the overall score, per category scores, and details. { “overall_score”: 0, “per_category_scores”: {}, “details”: [], “results”: [] }
- Return type:
Dict[str, Any]
- class graphdoc.prompts.DocQualitySignature(*, database_schema: str, category: Literal['perfect', 'almost perfect', 'poor but correct', 'incorrect'], rating: Literal[4, 3, 2, 1])[source]
Bases:
SignatureYou are a documentation quality evaluator specializing in GraphQL schemas. Your task is to assess the quality of documentation provided for a given database schema. Carefully analyze the schema’s descriptions for clarity, accuracy, and completeness. Categorize the documentation into one of the following ratings based on your evaluation: - perfect (4): The documentation is comprehensive and leaves no room for ambiguity in understanding the schema and its database content. - almost perfect (3): The documentation is clear and mostly free of ambiguity, but there is potential for further improvement. - poor but correct (2): The documentation is correct but lacks detail, resulting in some ambiguity. It requires enhancement to be more informative. - incorrect (1): The documentation contains errors or misleading information, regardless of any correct segments present. Such inaccuracies necessitate an incorrect rating. Provide a step-by-step reasoning to support your evaluation, along with the appropriate category label and numerical rating.
- database_schema: str
- category: Literal['perfect', 'almost perfect', 'poor but correct', 'incorrect']
- rating: Literal[4, 3, 2, 1]
- model_config: ClassVar[ConfigDict] = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class graphdoc.prompts.PromptFactory[source]
Bases:
object- static single_prompt(prompt: str | Signature | SignatureMeta, prompt_class: str, prompt_type: str, prompt_metric: str | DocQualityPrompt | SinglePrompt) SinglePrompt[source]
Returns an instance of the specified prompt class. Allows for the user to pass in their own dspy signature.
- Parameters:
- Returns:
An instance of the specified prompt class.
- Return type:
SinglePrompt
- class graphdoc.prompts.SinglePrompt(prompt: Signature | SignatureMeta, prompt_type: Literal['predict', 'chain_of_thought'] | Callable, prompt_metric: Any)[source]
Bases:
ABC- __init__(prompt: Signature | SignatureMeta, prompt_type: Literal['predict', 'chain_of_thought'] | Callable, prompt_metric: Any) None[source]
Initialize a single prompt.
- Parameters:
prompt (dspy.Signature) – The prompt to use.
prompt_type (Union[Literal["predict", "chain_of_thought"], Callable]) – The type of prompt to use. Can be “predict” or “chain_of_thought”. Optionally, pass another dspy.Module.
prompt_metric (Any) – The metric to use. Marked as Any for flexibility (as metrics can be other prompts).
- abstract evaluate_metric(example: Example, prediction: Prediction, trace=None) Any[source]
This is the metric used to evalaute the prompt.
- Parameters:
example (dspy.Example) – The example to evaluate the metric on.
prediction (dspy.Prediction) – The prediction to evaluate the metric on.
trace (Any) – The trace to evaluate the metric on. This is for DSPy.
- abstract format_metric(examples: List[Example], overall_score: float, results: List, scores: List) Dict[str, Any][source]
This takes the results from the evaluate_evalset and does any necessary formatting, taking into account the metric type.
- Parameters:
examples (List[dspy.Example]) – The examples to evaluate the metric on.
overall_score (float) – The overall score of the metric.
results (List) – The results from the evaluate_evalset.
scores (List) – The scores from the evaluate_evalset.
- abstract compare_metrics(base_metrics: Any, optimized_metrics: Any, comparison_value: str = 'overall_score') bool[source]
Compare the metrics of the base and optimized models. Return true if the optimized model is better than the base model.
- Parameters:
base_metrics (Any) – The metrics of the base model.
optimized_metrics (Any) – The metrics of the optimized model.
comparison_value (str) – The value to compare the metrics on. Determines which metric is used to compare the models.
- Returns:
True if the optimized model is better than the base model, False otherwise.
- Return type:
- evaluate_evalset(examples: List[Example], num_threads: int = 1, display_progress: bool = True, display_table: bool = True) Dict[str, Any][source]
Take in a list of examples and evaluate the results.
- Parameters:
- Returns:
A dictionary containing the overall score, results, and scores.
- Return type:
Dict[str, Any]