llvm-capstone/clang-tools-extra/clangd/Quality.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

571 lines
20 KiB
C++
Raw Normal View History

//===--- Quality.cpp ---------------------------------------------*- C++-*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
#include "Quality.h"
#include "AST.h"
[clangd] Use Decision Forest to score code completions. By default clangd will score a code completion item using heuristics model. Scoring can be done by Decision Forest model by passing `--ranking_model=decision_forest` to clangd. Features omitted from the model: - `NameMatch` is excluded because the final score must be multiplicative in `NameMatch` to allow rescoring by the editor. - `NeedsFixIts` is excluded because the generating dataset that needs 'fixits' is non-trivial. There are multiple ways (heuristics) to combine the above two features with the prediction of the DF: - `NeedsFixIts` is used as is with a penalty of `0.5`. Various alternatives of combining NameMatch `N` and Decision forest Prediction `P` - N * scale(P, 0, 1): Linearly scale the output of model to range [0, 1] - N * a^P: - More natural: Prediction of each Decision Tree can be considered as a multiplicative boost (like NameMatch) - Ordering is independent of the absolute value of P. Order of two items is proportional to `a^{difference in model prediction score}`. Higher `a` gives higher weightage to model output as compared to NameMatch score. Baseline MRR = 0.619 MRR for various combinations: N * P = 0.6346, advantage%=2.5768 N * 1.1^P = 0.6600, advantage%=6.6853 N * **1.2**^P = 0.6669, advantage%=**7.8005** N * **1.3**^P = 0.6668, advantage%=**7.7795** N * **1.4**^P = 0.6659, advantage%=**7.6270** N * 1.5^P = 0.6646, advantage%=7.4200 N * 1.6^P = 0.6636, advantage%=7.2671 N * 1.7^P = 0.6629, advantage%=7.1450 N * 2^P = 0.6612, advantage%=6.8673 N * 2.5^P = 0.6598, advantage%=6.6491 N * 3^P = 0.6590, advantage%=6.5242 N * scaled[0, 1] = 0.6465, advantage%=4.5054 Differential Revision: https://reviews.llvm.org/D88281
2020-09-22 05:56:08 +00:00
#include "CompletionModel.h"
#include "FileDistance.h"
#include "SourceCode.h"
#include "URI.h"
#include "index/Symbol.h"
#include "clang/AST/ASTContext.h"
#include "clang/AST/Decl.h"
#include "clang/AST/DeclCXX.h"
#include "clang/AST/DeclTemplate.h"
#include "clang/AST/DeclVisitor.h"
#include "clang/Basic/CharInfo.h"
#include "clang/Basic/SourceManager.h"
#include "clang/Sema/CodeCompleteConsumer.h"
#include "llvm/ADT/ArrayRef.h"
#include "llvm/ADT/SmallString.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/Casting.h"
#include "llvm/Support/FormatVariadic.h"
#include "llvm/Support/MathExtras.h"
#include "llvm/Support/raw_ostream.h"
#include <algorithm>
#include <cmath>
namespace clang {
namespace clangd {
static bool isReserved(llvm::StringRef Name) {
// FIXME: Should we exclude _Bool and others recognized by the standard?
return Name.size() >= 2 && Name[0] == '_' &&
(isUppercase(Name[1]) || Name[1] == '_');
}
static bool hasDeclInMainFile(const Decl &D) {
auto &SourceMgr = D.getASTContext().getSourceManager();
for (auto *Redecl : D.redecls()) {
if (isInsideMainFile(Redecl->getLocation(), SourceMgr))
return true;
}
return false;
}
static bool hasUsingDeclInMainFile(const CodeCompletionResult &R) {
const auto &Context = R.Declaration->getASTContext();
const auto &SourceMgr = Context.getSourceManager();
if (R.ShadowDecl) {
if (isInsideMainFile(R.ShadowDecl->getLocation(), SourceMgr))
return true;
}
return false;
}
static SymbolQualitySignals::SymbolCategory categorize(const NamedDecl &ND) {
if (const auto *FD = dyn_cast<FunctionDecl>(&ND)) {
if (FD->isOverloadedOperator())
return SymbolQualitySignals::Operator;
}
class Switch
: public ConstDeclVisitor<Switch, SymbolQualitySignals::SymbolCategory> {
public:
#define MAP(DeclType, Category) \
SymbolQualitySignals::SymbolCategory Visit##DeclType(const DeclType *) { \
return SymbolQualitySignals::Category; \
}
MAP(NamespaceDecl, Namespace);
MAP(NamespaceAliasDecl, Namespace);
MAP(TypeDecl, Type);
MAP(TypeAliasTemplateDecl, Type);
MAP(ClassTemplateDecl, Type);
[clangd] Tune down quality score for class constructors so that it's ranked after class types. Summary: Currently, class constructors have the same score as the class types, and they are often ranked before class types. This is often not desireable and can be annoying when snippet is enabled and constructor signatures are added. Metrics: ``` ================================================================================================== OVERALL ================================================================================================== Total measurements: 111117 (+0) All measurements: MRR: 64.06 (+0.20) Top-5: 75.73% (+0.14%) Top-100: 93.71% (+0.01%) Full identifiers: MRR: 98.25 (+0.55) Top-5: 99.04% (+0.03%) Top-100: 99.16% (+0.00%) Filter length 0-5: MRR: 15.23 (+0.02) 50.50 (-0.02) 65.04 (+0.11) 70.75 (+0.19) 74.37 (+0.25) 79.43 (+0.32) Top-5: 40.90% (+0.03%) 74.52% (+0.03%) 87.23% (+0.15%) 91.68% (+0.08%) 93.68% (+0.14%) 95.87% (+0.12%) Top-100: 68.21% (+0.02%) 96.28% (+0.07%) 98.43% (+0.00%) 98.72% (+0.00%) 98.74% (+0.01%) 98.81% (+0.00%) ================================================================================================== DEFAULT ================================================================================================== Total measurements: 57535 (+0) All measurements: MRR: 58.07 (+0.37) Top-5: 69.94% (+0.26%) Top-100: 90.14% (+0.03%) Full identifiers: MRR: 97.13 (+1.05) Top-5: 98.14% (+0.06%) Top-100: 98.34% (+0.00%) Filter length 0-5: MRR: 13.91 (+0.00) 38.53 (+0.01) 55.58 (+0.21) 63.63 (+0.30) 69.23 (+0.47) 72.87 (+0.60) Top-5: 24.99% (+0.00%) 62.70% (+0.06%) 82.80% (+0.30%) 88.66% (+0.16%) 92.02% (+0.27%) 93.53% (+0.21%) Top-100: 51.56% (+0.05%) 93.19% (+0.13%) 97.30% (+0.00%) 97.81% (+0.00%) 97.85% (+0.01%) 97.79% (+0.00%) ``` Remark: - The full-id completions have +1.05 MRR improvement. - There is no noticeable impact on EXPLICIT_MEMBER_ACCESS and WANT_LOCAL. Reviewers: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D49667 llvm-svn: 337816
2018-07-24 08:51:52 +00:00
MAP(CXXConstructorDecl, Constructor);
MAP(CXXDestructorDecl, Destructor);
MAP(ValueDecl, Variable);
MAP(VarTemplateDecl, Variable);
MAP(FunctionDecl, Function);
MAP(FunctionTemplateDecl, Function);
MAP(Decl, Unknown);
#undef MAP
};
return Switch().Visit(&ND);
}
static SymbolQualitySignals::SymbolCategory
categorize(const CodeCompletionResult &R) {
if (R.Declaration)
return categorize(*R.Declaration);
if (R.Kind == CodeCompletionResult::RK_Macro)
return SymbolQualitySignals::Macro;
// Everything else is a keyword or a pattern. Patterns are mostly keywords
// too, except a few which we recognize by cursor kind.
switch (R.CursorKind) {
case CXCursor_CXXMethod:
return SymbolQualitySignals::Function;
case CXCursor_ModuleImportDecl:
return SymbolQualitySignals::Namespace;
case CXCursor_MacroDefinition:
return SymbolQualitySignals::Macro;
case CXCursor_TypeRef:
return SymbolQualitySignals::Type;
case CXCursor_MemberRef:
return SymbolQualitySignals::Variable;
[clangd] Tune down quality score for class constructors so that it's ranked after class types. Summary: Currently, class constructors have the same score as the class types, and they are often ranked before class types. This is often not desireable and can be annoying when snippet is enabled and constructor signatures are added. Metrics: ``` ================================================================================================== OVERALL ================================================================================================== Total measurements: 111117 (+0) All measurements: MRR: 64.06 (+0.20) Top-5: 75.73% (+0.14%) Top-100: 93.71% (+0.01%) Full identifiers: MRR: 98.25 (+0.55) Top-5: 99.04% (+0.03%) Top-100: 99.16% (+0.00%) Filter length 0-5: MRR: 15.23 (+0.02) 50.50 (-0.02) 65.04 (+0.11) 70.75 (+0.19) 74.37 (+0.25) 79.43 (+0.32) Top-5: 40.90% (+0.03%) 74.52% (+0.03%) 87.23% (+0.15%) 91.68% (+0.08%) 93.68% (+0.14%) 95.87% (+0.12%) Top-100: 68.21% (+0.02%) 96.28% (+0.07%) 98.43% (+0.00%) 98.72% (+0.00%) 98.74% (+0.01%) 98.81% (+0.00%) ================================================================================================== DEFAULT ================================================================================================== Total measurements: 57535 (+0) All measurements: MRR: 58.07 (+0.37) Top-5: 69.94% (+0.26%) Top-100: 90.14% (+0.03%) Full identifiers: MRR: 97.13 (+1.05) Top-5: 98.14% (+0.06%) Top-100: 98.34% (+0.00%) Filter length 0-5: MRR: 13.91 (+0.00) 38.53 (+0.01) 55.58 (+0.21) 63.63 (+0.30) 69.23 (+0.47) 72.87 (+0.60) Top-5: 24.99% (+0.00%) 62.70% (+0.06%) 82.80% (+0.30%) 88.66% (+0.16%) 92.02% (+0.27%) 93.53% (+0.21%) Top-100: 51.56% (+0.05%) 93.19% (+0.13%) 97.30% (+0.00%) 97.81% (+0.00%) 97.85% (+0.01%) 97.79% (+0.00%) ``` Remark: - The full-id completions have +1.05 MRR improvement. - There is no noticeable impact on EXPLICIT_MEMBER_ACCESS and WANT_LOCAL. Reviewers: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D49667 llvm-svn: 337816
2018-07-24 08:51:52 +00:00
case CXCursor_Constructor:
return SymbolQualitySignals::Constructor;
default:
return SymbolQualitySignals::Keyword;
}
}
static SymbolQualitySignals::SymbolCategory
categorize(const index::SymbolInfo &D) {
switch (D.Kind) {
case index::SymbolKind::Namespace:
case index::SymbolKind::NamespaceAlias:
return SymbolQualitySignals::Namespace;
case index::SymbolKind::Macro:
return SymbolQualitySignals::Macro;
case index::SymbolKind::Enum:
case index::SymbolKind::Struct:
case index::SymbolKind::Class:
case index::SymbolKind::Protocol:
case index::SymbolKind::Extension:
case index::SymbolKind::Union:
case index::SymbolKind::TypeAlias:
case index::SymbolKind::TemplateTypeParm:
case index::SymbolKind::TemplateTemplateParm:
return SymbolQualitySignals::Type;
case index::SymbolKind::Function:
case index::SymbolKind::ClassMethod:
case index::SymbolKind::InstanceMethod:
case index::SymbolKind::StaticMethod:
case index::SymbolKind::InstanceProperty:
case index::SymbolKind::ClassProperty:
case index::SymbolKind::StaticProperty:
case index::SymbolKind::ConversionFunction:
return SymbolQualitySignals::Function;
case index::SymbolKind::Destructor:
return SymbolQualitySignals::Destructor;
[clangd] Tune down quality score for class constructors so that it's ranked after class types. Summary: Currently, class constructors have the same score as the class types, and they are often ranked before class types. This is often not desireable and can be annoying when snippet is enabled and constructor signatures are added. Metrics: ``` ================================================================================================== OVERALL ================================================================================================== Total measurements: 111117 (+0) All measurements: MRR: 64.06 (+0.20) Top-5: 75.73% (+0.14%) Top-100: 93.71% (+0.01%) Full identifiers: MRR: 98.25 (+0.55) Top-5: 99.04% (+0.03%) Top-100: 99.16% (+0.00%) Filter length 0-5: MRR: 15.23 (+0.02) 50.50 (-0.02) 65.04 (+0.11) 70.75 (+0.19) 74.37 (+0.25) 79.43 (+0.32) Top-5: 40.90% (+0.03%) 74.52% (+0.03%) 87.23% (+0.15%) 91.68% (+0.08%) 93.68% (+0.14%) 95.87% (+0.12%) Top-100: 68.21% (+0.02%) 96.28% (+0.07%) 98.43% (+0.00%) 98.72% (+0.00%) 98.74% (+0.01%) 98.81% (+0.00%) ================================================================================================== DEFAULT ================================================================================================== Total measurements: 57535 (+0) All measurements: MRR: 58.07 (+0.37) Top-5: 69.94% (+0.26%) Top-100: 90.14% (+0.03%) Full identifiers: MRR: 97.13 (+1.05) Top-5: 98.14% (+0.06%) Top-100: 98.34% (+0.00%) Filter length 0-5: MRR: 13.91 (+0.00) 38.53 (+0.01) 55.58 (+0.21) 63.63 (+0.30) 69.23 (+0.47) 72.87 (+0.60) Top-5: 24.99% (+0.00%) 62.70% (+0.06%) 82.80% (+0.30%) 88.66% (+0.16%) 92.02% (+0.27%) 93.53% (+0.21%) Top-100: 51.56% (+0.05%) 93.19% (+0.13%) 97.30% (+0.00%) 97.81% (+0.00%) 97.85% (+0.01%) 97.79% (+0.00%) ``` Remark: - The full-id completions have +1.05 MRR improvement. - There is no noticeable impact on EXPLICIT_MEMBER_ACCESS and WANT_LOCAL. Reviewers: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D49667 llvm-svn: 337816
2018-07-24 08:51:52 +00:00
case index::SymbolKind::Constructor:
return SymbolQualitySignals::Constructor;
case index::SymbolKind::Variable:
case index::SymbolKind::Field:
case index::SymbolKind::EnumConstant:
case index::SymbolKind::Parameter:
case index::SymbolKind::NonTypeTemplateParm:
return SymbolQualitySignals::Variable;
case index::SymbolKind::Using:
case index::SymbolKind::Module:
case index::SymbolKind::Unknown:
return SymbolQualitySignals::Unknown;
}
llvm_unreachable("Unknown index::SymbolKind");
}
static bool isInstanceMember(const NamedDecl *ND) {
if (!ND)
return false;
if (const auto *TP = dyn_cast<FunctionTemplateDecl>(ND))
ND = TP->TemplateDecl::getTemplatedDecl();
if (const auto *CM = dyn_cast<CXXMethodDecl>(ND))
return !CM->isStatic();
return isa<FieldDecl>(ND); // Note that static fields are VarDecl.
}
static bool isInstanceMember(const index::SymbolInfo &D) {
switch (D.Kind) {
case index::SymbolKind::InstanceMethod:
case index::SymbolKind::InstanceProperty:
case index::SymbolKind::Field:
return true;
default:
return false;
}
}
void SymbolQualitySignals::merge(const CodeCompletionResult &SemaCCResult) {
Deprecated |= (SemaCCResult.Availability == CXAvailability_Deprecated);
Category = categorize(SemaCCResult);
if (SemaCCResult.Declaration) {
ImplementationDetail |= isImplementationDetail(SemaCCResult.Declaration);
if (auto *ID = SemaCCResult.Declaration->getIdentifier())
ReservedName = ReservedName || isReserved(ID->getName());
} else if (SemaCCResult.Kind == CodeCompletionResult::RK_Macro)
ReservedName = ReservedName || isReserved(SemaCCResult.Macro->getName());
}
void SymbolQualitySignals::merge(const Symbol &IndexResult) {
Deprecated |= (IndexResult.Flags & Symbol::Deprecated);
ImplementationDetail |= (IndexResult.Flags & Symbol::ImplementationDetail);
References = std::max(IndexResult.References, References);
Category = categorize(IndexResult.SymInfo);
ReservedName = ReservedName || isReserved(IndexResult.Name);
}
float SymbolQualitySignals::evaluateHeuristics() const {
float Score = 1;
// This avoids a sharp gradient for tail symbols, and also neatly avoids the
// question of whether 0 references means a bad symbol or missing data.
if (References >= 10) {
// Use a sigmoid style boosting function, which flats out nicely for large
// numbers (e.g. 2.58 for 1M references).
// The following boosting function is equivalent to:
// m = 0.06
// f = 12.0
// boost = f * sigmoid(m * std::log(References)) - 0.5 * f + 0.59
// Sample data points: (10, 1.00), (100, 1.41), (1000, 1.82),
// (10K, 2.21), (100K, 2.58), (1M, 2.94)
float S = std::pow(References, -0.06);
Score *= 6.0 * (1 - S) / (1 + S) + 0.59;
}
if (Deprecated)
Score *= 0.1f;
if (ReservedName)
Score *= 0.1f;
if (ImplementationDetail)
Score *= 0.2f;
switch (Category) {
case Keyword: // Often relevant, but misses most signals.
Score *= 4; // FIXME: important keywords should have specific boosts.
break;
case Type:
case Function:
case Variable:
Score *= 1.1f;
break;
case Namespace:
Score *= 0.8f;
break;
case Macro:
case Destructor:
case Operator:
Score *= 0.5f;
break;
[clangd] Tune down quality score for class constructors so that it's ranked after class types. Summary: Currently, class constructors have the same score as the class types, and they are often ranked before class types. This is often not desireable and can be annoying when snippet is enabled and constructor signatures are added. Metrics: ``` ================================================================================================== OVERALL ================================================================================================== Total measurements: 111117 (+0) All measurements: MRR: 64.06 (+0.20) Top-5: 75.73% (+0.14%) Top-100: 93.71% (+0.01%) Full identifiers: MRR: 98.25 (+0.55) Top-5: 99.04% (+0.03%) Top-100: 99.16% (+0.00%) Filter length 0-5: MRR: 15.23 (+0.02) 50.50 (-0.02) 65.04 (+0.11) 70.75 (+0.19) 74.37 (+0.25) 79.43 (+0.32) Top-5: 40.90% (+0.03%) 74.52% (+0.03%) 87.23% (+0.15%) 91.68% (+0.08%) 93.68% (+0.14%) 95.87% (+0.12%) Top-100: 68.21% (+0.02%) 96.28% (+0.07%) 98.43% (+0.00%) 98.72% (+0.00%) 98.74% (+0.01%) 98.81% (+0.00%) ================================================================================================== DEFAULT ================================================================================================== Total measurements: 57535 (+0) All measurements: MRR: 58.07 (+0.37) Top-5: 69.94% (+0.26%) Top-100: 90.14% (+0.03%) Full identifiers: MRR: 97.13 (+1.05) Top-5: 98.14% (+0.06%) Top-100: 98.34% (+0.00%) Filter length 0-5: MRR: 13.91 (+0.00) 38.53 (+0.01) 55.58 (+0.21) 63.63 (+0.30) 69.23 (+0.47) 72.87 (+0.60) Top-5: 24.99% (+0.00%) 62.70% (+0.06%) 82.80% (+0.30%) 88.66% (+0.16%) 92.02% (+0.27%) 93.53% (+0.21%) Top-100: 51.56% (+0.05%) 93.19% (+0.13%) 97.30% (+0.00%) 97.81% (+0.00%) 97.85% (+0.01%) 97.79% (+0.00%) ``` Remark: - The full-id completions have +1.05 MRR improvement. - There is no noticeable impact on EXPLICIT_MEMBER_ACCESS and WANT_LOCAL. Reviewers: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D49667 llvm-svn: 337816
2018-07-24 08:51:52 +00:00
case Constructor: // No boost constructors so they are after class types.
case Unknown:
break;
}
return Score;
}
llvm::raw_ostream &operator<<(llvm::raw_ostream &OS,
const SymbolQualitySignals &S) {
OS << llvm::formatv("=== Symbol quality: {0}\n", S.evaluateHeuristics());
OS << llvm::formatv("\tReferences: {0}\n", S.References);
OS << llvm::formatv("\tDeprecated: {0}\n", S.Deprecated);
OS << llvm::formatv("\tReserved name: {0}\n", S.ReservedName);
OS << llvm::formatv("\tImplementation detail: {0}\n", S.ImplementationDetail);
OS << llvm::formatv("\tCategory: {0}\n", static_cast<int>(S.Category));
return OS;
}
static SymbolRelevanceSignals::AccessibleScope
computeScope(const NamedDecl *D) {
// Injected "Foo" within the class "Foo" has file scope, not class scope.
const DeclContext *DC = D->getDeclContext();
if (auto *R = dyn_cast_or_null<RecordDecl>(D))
if (R->isInjectedClassName())
DC = DC->getParent();
// Class constructor should have the same scope as the class.
if (isa<CXXConstructorDecl>(D))
DC = DC->getParent();
bool InClass = false;
for (; !DC->isFileContext(); DC = DC->getParent()) {
if (DC->isFunctionOrMethod())
return SymbolRelevanceSignals::FunctionScope;
InClass = InClass || DC->isRecord();
}
if (InClass)
return SymbolRelevanceSignals::ClassScope;
// ExternalLinkage threshold could be tweaked, e.g. module-visible as global.
// Avoid caching linkage if it may change after enclosing code completion.
if (hasUnstableLinkage(D) || D->getLinkageInternal() < ExternalLinkage)
return SymbolRelevanceSignals::FileScope;
return SymbolRelevanceSignals::GlobalScope;
}
void SymbolRelevanceSignals::merge(const Symbol &IndexResult) {
SymbolURI = IndexResult.CanonicalDeclaration.FileURI;
SymbolScope = IndexResult.Scope;
IsInstanceMember |= isInstanceMember(IndexResult.SymInfo);
if (!(IndexResult.Flags & Symbol::VisibleOutsideFile)) {
Scope = AccessibleScope::FileScope;
}
}
void SymbolRelevanceSignals::merge(const CodeCompletionResult &SemaCCResult) {
if (SemaCCResult.Availability == CXAvailability_NotAvailable ||
SemaCCResult.Availability == CXAvailability_NotAccessible)
Forbidden = true;
if (SemaCCResult.Declaration) {
SemaSaysInScope = true;
// We boost things that have decls in the main file. We give a fixed score
// for all other declarations in sema as they are already included in the
// translation unit.
float DeclProximity = (hasDeclInMainFile(*SemaCCResult.Declaration) ||
hasUsingDeclInMainFile(SemaCCResult))
? 1.0
: 0.6;
SemaFileProximityScore = std::max(DeclProximity, SemaFileProximityScore);
IsInstanceMember |= isInstanceMember(SemaCCResult.Declaration);
InBaseClass |= SemaCCResult.InBaseClass;
}
// Declarations are scoped, others (like macros) are assumed global.
if (SemaCCResult.Declaration)
Scope = std::min(Scope, computeScope(SemaCCResult.Declaration));
NeedsFixIts = !SemaCCResult.FixIts.empty();
}
static float fileProximityScore(unsigned FileDistance) {
// Range: [0, 1]
// FileDistance = [0, 1, 2, 3, 4, .., FileDistance::Unreachable]
// Score = [1, 0.82, 0.67, 0.55, 0.45, .., 0]
if (FileDistance == FileDistance::Unreachable)
return 0;
// Assume approximately default options are used for sensible scoring.
return std::exp(FileDistance * -0.4f / FileDistanceOptions().UpCost);
}
static float scopeProximityScore(unsigned ScopeDistance) {
// Range: [0.6, 2].
// ScopeDistance = [0, 1, 2, 3, 4, 5, 6, 7, .., FileDistance::Unreachable]
// Score = [2.0, 1.55, 1.2, 0.93, 0.72, 0.65, 0.65, 0.65, .., 0.6]
if (ScopeDistance == FileDistance::Unreachable)
return 0.6f;
return std::max(0.65, 2.0 * std::pow(0.6, ScopeDistance / 2.0));
}
static llvm::Optional<llvm::StringRef>
wordMatching(llvm::StringRef Name, const llvm::StringSet<> *ContextWords) {
if (ContextWords)
for (const auto &Word : ContextWords->keys())
if (Name.contains_lower(Word))
return Word;
return llvm::None;
}
SymbolRelevanceSignals::DerivedSignals
SymbolRelevanceSignals::calculateDerivedSignals() const {
DerivedSignals Derived;
Derived.NameMatchesContext = wordMatching(Name, ContextWords).hasValue();
Derived.FileProximityDistance = !FileProximityMatch || SymbolURI.empty()
? FileDistance::Unreachable
: FileProximityMatch->distance(SymbolURI);
if (ScopeProximityMatch) {
// For global symbol, the distance is 0.
Derived.ScopeProximityDistance =
SymbolScope ? ScopeProximityMatch->distance(*SymbolScope) : 0;
}
return Derived;
}
float SymbolRelevanceSignals::evaluateHeuristics() const {
DerivedSignals Derived = calculateDerivedSignals();
float Score = 1;
if (Forbidden)
return 0;
Score *= NameMatch;
// File proximity scores are [0,1] and we translate them into a multiplier in
// the range from 1 to 3.
Score *= 1 + 2 * std::max(fileProximityScore(Derived.FileProximityDistance),
SemaFileProximityScore);
if (ScopeProximityMatch)
// Use a constant scope boost for sema results, as scopes of sema results
// can be tricky (e.g. class/function scope). Set to the max boost as we
// don't load top-level symbols from the preamble and sema results are
// always in the accessible scope.
Score *= SemaSaysInScope
? 2.0
: scopeProximityScore(Derived.ScopeProximityDistance);
if (Derived.NameMatchesContext)
Score *= 1.5;
// Symbols like local variables may only be referenced within their scope.
// Conversely if we're in that scope, it's likely we'll reference them.
if (Query == CodeComplete) {
// The narrower the scope where a symbol is visible, the more likely it is
// to be relevant when it is available.
switch (Scope) {
case GlobalScope:
break;
case FileScope:
Score *= 1.5f;
break;
case ClassScope:
Score *= 2;
break;
case FunctionScope:
Score *= 4;
break;
}
} else {
// For non-completion queries, the wider the scope where a symbol is
// visible, the more likely it is to be relevant.
switch (Scope) {
case GlobalScope:
break;
case FileScope:
Score *= 0.5f;
break;
default:
// TODO: Handle other scopes as we start to use them for index results.
break;
}
}
if (TypeMatchesPreferred)
Score *= 5.0;
// Penalize non-instance members when they are accessed via a class instance.
if (!IsInstanceMember &&
(Context == CodeCompletionContext::CCC_DotMemberAccess ||
Context == CodeCompletionContext::CCC_ArrowMemberAccess)) {
Score *= 0.2f;
}
if (InBaseClass)
Score *= 0.5f;
// Penalize for FixIts.
if (NeedsFixIts)
Score *= 0.5f;
return Score;
}
llvm::raw_ostream &operator<<(llvm::raw_ostream &OS,
const SymbolRelevanceSignals &S) {
OS << llvm::formatv("=== Symbol relevance: {0}\n", S.evaluateHeuristics());
OS << llvm::formatv("\tName: {0}\n", S.Name);
OS << llvm::formatv("\tName match: {0}\n", S.NameMatch);
if (S.ContextWords)
OS << llvm::formatv(
"\tMatching context word: {0}\n",
wordMatching(S.Name, S.ContextWords).getValueOr("<none>"));
OS << llvm::formatv("\tForbidden: {0}\n", S.Forbidden);
OS << llvm::formatv("\tNeedsFixIts: {0}\n", S.NeedsFixIts);
OS << llvm::formatv("\tIsInstanceMember: {0}\n", S.IsInstanceMember);
OS << llvm::formatv("\tInBaseClass: {0}\n", S.InBaseClass);
OS << llvm::formatv("\tContext: {0}\n", getCompletionKindString(S.Context));
OS << llvm::formatv("\tQuery type: {0}\n", static_cast<int>(S.Query));
OS << llvm::formatv("\tScope: {0}\n", static_cast<int>(S.Scope));
OS << llvm::formatv("\tSymbol URI: {0}\n", S.SymbolURI);
OS << llvm::formatv("\tSymbol scope: {0}\n",
S.SymbolScope ? *S.SymbolScope : "<None>");
SymbolRelevanceSignals::DerivedSignals Derived = S.calculateDerivedSignals();
if (S.FileProximityMatch) {
unsigned Score = fileProximityScore(Derived.FileProximityDistance);
OS << llvm::formatv("\tIndex URI proximity: {0} (distance={1})\n", Score,
Derived.FileProximityDistance);
}
OS << llvm::formatv("\tSema file proximity: {0}\n", S.SemaFileProximityScore);
OS << llvm::formatv("\tSema says in scope: {0}\n", S.SemaSaysInScope);
if (S.ScopeProximityMatch)
OS << llvm::formatv("\tIndex scope boost: {0}\n",
scopeProximityScore(Derived.ScopeProximityDistance));
OS << llvm::formatv(
"\tType matched preferred: {0} (Context type: {1}, Symbol type: {2}\n",
S.TypeMatchesPreferred, S.HadContextType, S.HadSymbolType);
return OS;
}
float evaluateSymbolAndRelevance(float SymbolQuality, float SymbolRelevance) {
return SymbolQuality * SymbolRelevance;
}
DecisionForestScores
evaluateDecisionForest(const SymbolQualitySignals &Quality,
const SymbolRelevanceSignals &Relevance, float Base) {
[clangd] Use Decision Forest to score code completions. By default clangd will score a code completion item using heuristics model. Scoring can be done by Decision Forest model by passing `--ranking_model=decision_forest` to clangd. Features omitted from the model: - `NameMatch` is excluded because the final score must be multiplicative in `NameMatch` to allow rescoring by the editor. - `NeedsFixIts` is excluded because the generating dataset that needs 'fixits' is non-trivial. There are multiple ways (heuristics) to combine the above two features with the prediction of the DF: - `NeedsFixIts` is used as is with a penalty of `0.5`. Various alternatives of combining NameMatch `N` and Decision forest Prediction `P` - N * scale(P, 0, 1): Linearly scale the output of model to range [0, 1] - N * a^P: - More natural: Prediction of each Decision Tree can be considered as a multiplicative boost (like NameMatch) - Ordering is independent of the absolute value of P. Order of two items is proportional to `a^{difference in model prediction score}`. Higher `a` gives higher weightage to model output as compared to NameMatch score. Baseline MRR = 0.619 MRR for various combinations: N * P = 0.6346, advantage%=2.5768 N * 1.1^P = 0.6600, advantage%=6.6853 N * **1.2**^P = 0.6669, advantage%=**7.8005** N * **1.3**^P = 0.6668, advantage%=**7.7795** N * **1.4**^P = 0.6659, advantage%=**7.6270** N * 1.5^P = 0.6646, advantage%=7.4200 N * 1.6^P = 0.6636, advantage%=7.2671 N * 1.7^P = 0.6629, advantage%=7.1450 N * 2^P = 0.6612, advantage%=6.8673 N * 2.5^P = 0.6598, advantage%=6.6491 N * 3^P = 0.6590, advantage%=6.5242 N * scaled[0, 1] = 0.6465, advantage%=4.5054 Differential Revision: https://reviews.llvm.org/D88281
2020-09-22 05:56:08 +00:00
Example E;
E.setIsDeprecated(Quality.Deprecated);
E.setIsReservedName(Quality.ReservedName);
E.setIsImplementationDetail(Quality.ImplementationDetail);
E.setNumReferences(Quality.References);
E.setSymbolCategory(Quality.Category);
SymbolRelevanceSignals::DerivedSignals Derived =
Relevance.calculateDerivedSignals();
E.setIsNameInContext(Derived.NameMatchesContext);
E.setIsForbidden(Relevance.Forbidden);
E.setIsInBaseClass(Relevance.InBaseClass);
E.setFileProximityDistance(Derived.FileProximityDistance);
E.setSemaFileProximityScore(Relevance.SemaFileProximityScore);
E.setSymbolScopeDistance(Derived.ScopeProximityDistance);
E.setSemaSaysInScope(Relevance.SemaSaysInScope);
E.setScope(Relevance.Scope);
E.setContextKind(Relevance.Context);
E.setIsInstanceMember(Relevance.IsInstanceMember);
E.setHadContextType(Relevance.HadContextType);
E.setHadSymbolType(Relevance.HadSymbolType);
E.setTypeMatchesPreferred(Relevance.TypeMatchesPreferred);
E.setFilterLength(Relevance.FilterLength);
DecisionForestScores Scores;
// Exponentiating DecisionForest prediction makes the score of each tree a
// multiplciative boost (like NameMatch). This allows us to weigh the
// prediciton score and NameMatch appropriately.
Scores.ExcludingName = pow(Base, Evaluate(E));
// NeedsFixIts is not part of the DecisionForest as generating training
// data that needs fixits is not-feasible.
if (Relevance.NeedsFixIts)
Scores.ExcludingName *= 0.5;
// NameMatch should be a multiplier on total score to support rescoring.
Scores.Total = Relevance.NameMatch * Scores.ExcludingName;
return Scores;
[clangd] Use Decision Forest to score code completions. By default clangd will score a code completion item using heuristics model. Scoring can be done by Decision Forest model by passing `--ranking_model=decision_forest` to clangd. Features omitted from the model: - `NameMatch` is excluded because the final score must be multiplicative in `NameMatch` to allow rescoring by the editor. - `NeedsFixIts` is excluded because the generating dataset that needs 'fixits' is non-trivial. There are multiple ways (heuristics) to combine the above two features with the prediction of the DF: - `NeedsFixIts` is used as is with a penalty of `0.5`. Various alternatives of combining NameMatch `N` and Decision forest Prediction `P` - N * scale(P, 0, 1): Linearly scale the output of model to range [0, 1] - N * a^P: - More natural: Prediction of each Decision Tree can be considered as a multiplicative boost (like NameMatch) - Ordering is independent of the absolute value of P. Order of two items is proportional to `a^{difference in model prediction score}`. Higher `a` gives higher weightage to model output as compared to NameMatch score. Baseline MRR = 0.619 MRR for various combinations: N * P = 0.6346, advantage%=2.5768 N * 1.1^P = 0.6600, advantage%=6.6853 N * **1.2**^P = 0.6669, advantage%=**7.8005** N * **1.3**^P = 0.6668, advantage%=**7.7795** N * **1.4**^P = 0.6659, advantage%=**7.6270** N * 1.5^P = 0.6646, advantage%=7.4200 N * 1.6^P = 0.6636, advantage%=7.2671 N * 1.7^P = 0.6629, advantage%=7.1450 N * 2^P = 0.6612, advantage%=6.8673 N * 2.5^P = 0.6598, advantage%=6.6491 N * 3^P = 0.6590, advantage%=6.5242 N * scaled[0, 1] = 0.6465, advantage%=4.5054 Differential Revision: https://reviews.llvm.org/D88281
2020-09-22 05:56:08 +00:00
}
// Produces an integer that sorts in the same order as F.
// That is: a < b <==> encodeFloat(a) < encodeFloat(b).
static uint32_t encodeFloat(float F) {
static_assert(std::numeric_limits<float>::is_iec559, "");
constexpr uint32_t TopBit = ~(~uint32_t{0} >> 1);
// Get the bits of the float. Endianness is the same as for integers.
uint32_t U = llvm::FloatToBits(F);
// IEEE 754 floats compare like sign-magnitude integers.
if (U & TopBit) // Negative float.
return 0 - U; // Map onto the low half of integers, order reversed.
return U + TopBit; // Positive floats map onto the high half of integers.
}
std::string sortText(float Score, llvm::StringRef Name) {
// We convert -Score to an integer, and hex-encode for readability.
// Example: [0.5, "foo"] -> "41000000foo"
std::string S;
llvm::raw_string_ostream OS(S);
llvm::write_hex(OS, encodeFloat(-Score), llvm::HexPrintStyle::Lower,
/*Width=*/2 * sizeof(Score));
OS << Name;
OS.flush();
return S;
}
llvm::raw_ostream &operator<<(llvm::raw_ostream &OS,
const SignatureQualitySignals &S) {
OS << llvm::formatv("=== Signature Quality:\n");
OS << llvm::formatv("\tNumber of parameters: {0}\n", S.NumberOfParameters);
OS << llvm::formatv("\tNumber of optional parameters: {0}\n",
S.NumberOfOptionalParameters);
OS << llvm::formatv("\tKind: {0}\n", S.Kind);
return OS;
}
} // namespace clangd
} // namespace clang